[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:09 dist-test-slave-7511 krb5kdc[4931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:09 dist-test-slave-7511 krb5kdc[4931](info): set up 2 sockets
Sep 23 22:17:09 dist-test-slave-7511 krb5kdc[4931](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 22:17:10.285615 4924 mini_kdc.cc:121] Time spent starting KDC: real 2.675s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260923 22:17:10.440120 4924 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.154s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260923 22:17:10.766197 4924 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.326s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:10 dist-test-slave-7511 krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:10.875198 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:45729
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:45729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:10.955135 4944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:10.955420 4944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:10.955464 4944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:10.956702 4944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:10.956775 4944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:10.956801 4944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:10.956833 4944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:10.956863 4944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:10.958343 4944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:45729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:45729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:10.958601 4944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:10.958914 4944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:10.964246 4949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:10.964365 4944 server_base.cc:1034] running on GCE node
W20260923 22:17:10.964238 4952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:10.964239 4950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:10.964766 4944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:10.965083 4944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:10.966228 4944 hybrid_clock.cc:648] HybridClock initialized: now 1790201830966173 us; error 75 us; skew 500 ppm
Sep 23 22:17:10 dist-test-slave-7511 krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:10.975131 4944 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:10.975677 4944 webserver.cc:460] Webserver started at http://127.4.207.62:46383/ using document root <none> and password file <none>
I20260923 22:17:10.975908 4944 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:10.975956 4944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:10.976125 4944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:10.977208 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "2ed20673623443f1b12761861fd4006e"
format_stamp: "Formatted at 2026-09-23 22:17:10 on dist-test-slave-7511"
server_key: "1e41ab39ba0d973e0f4035462b58ed39"
server_key_iv: "a137aff649599ae759b4ee5ac4d6dcb6"
server_key_version: "encryptionkey@0"
I20260923 22:17:10.977610 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "2ed20673623443f1b12761861fd4006e"
format_stamp: "Formatted at 2026-09-23 22:17:10 on dist-test-slave-7511"
server_key: "1e41ab39ba0d973e0f4035462b58ed39"
server_key_iv: "a137aff649599ae759b4ee5ac4d6dcb6"
server_key_version: "encryptionkey@0"
I20260923 22:17:10.979068 4944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:10.979950 4959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:10.980190 4944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:10.980290 4944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "2ed20673623443f1b12761861fd4006e"
format_stamp: "Formatted at 2026-09-23 22:17:10 on dist-test-slave-7511"
server_key: "1e41ab39ba0d973e0f4035462b58ed39"
server_key_iv: "a137aff649599ae759b4ee5ac4d6dcb6"
server_key_version: "encryptionkey@0"
I20260923 22:17:10.980365 4944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:10.994752 4944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:10.997301 4944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:10.997480 4944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:11.001744 4944 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:45729
I20260923 22:17:11.002146 4944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:11.002122 5011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:45729 every 8 connection(s)
I20260923 22:17:11.002827 5012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:11.005254 5012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e: Bootstrap starting.
I20260923 22:17:11.005856 5012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:11.006148 5012 log.cc:824] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:11.006831 5012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e: No bootstrap required, opened a new log
I20260923 22:17:11.008534 5012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed20673623443f1b12761861fd4006e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45729 } }
I20260923 22:17:11.008703 5012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:11.008760 5012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed20673623443f1b12761861fd4006e, State: Initialized, Role: FOLLOWER
I20260923 22:17:11.008940 5012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed20673623443f1b12761861fd4006e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45729 } }
I20260923 22:17:11.009027 5012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:11.009071 5012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:11.009141 5012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:11.009693 5012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed20673623443f1b12761861fd4006e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45729 } }
I20260923 22:17:11.009816 5012 leader_election.cc:302] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ed20673623443f1b12761861fd4006e; no voters:
I20260923 22:17:11.010044 5012 leader_election.cc:288] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:11.010075 5015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:11.010188 5015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [term 1 LEADER]: Becoming Leader. State: Replica: 2ed20673623443f1b12761861fd4006e, State: Running, Role: LEADER
I20260923 22:17:11.010316 5012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:11.010345 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 4944
I20260923 22:17:11.010330 5015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed20673623443f1b12761861fd4006e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45729 } }
I20260923 22:17:11.010457 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:11.010691 5017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ed20673623443f1b12761861fd4006e. Latest consensus state: current_term: 1 leader_uuid: "2ed20673623443f1b12761861fd4006e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed20673623443f1b12761861fd4006e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45729 } } }
I20260923 22:17:11.010766 5017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:11.010828 5016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ed20673623443f1b12761861fd4006e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed20673623443f1b12761861fd4006e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45729 } } }
I20260923 22:17:11.010907 5016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:11.010984 5021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:11.011356 4924 external_mini_cluster.cc:1428] Setting key 346b81139027bd14256a1f6c0172c713
I20260923 22:17:11.011618 5021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:11.013464 5021 catalog_manager.cc:1347] Generated new cluster ID: d42d0660bdba401f8b04bfcc6f276aa7
I20260923 22:17:11.013521 5021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:11.030501 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.012425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.4.207.62:45729)
0923 22:17:11.013035 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:11.013045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:11.013086 (+ 41us) server_negotiation.cc:407] Connection header received
0923 22:17:11.015386 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:11.015393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:11.015487 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:11.015632 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:11.017736 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.019300 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.020864 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.021515 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.021627 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.026121 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:11.026125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:11.026126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:11.026138 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:11.028306 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.028859 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.028862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.028862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.029001 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.029280 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.029281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.029281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.029626 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:11.029762 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:11.030093 (+ 331us) server_negotiation.cc:299] Negotiation successful
0923 22:17:11.030239 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":70,"threads_started":1}
I20260923 22:17:11.048516 5021 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:11.049278 5021 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:11.056632 5021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ed20673623443f1b12761861fd4006e: Generated new TSK 0
I20260923 22:17:11.056852 5021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:11.086789 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45729
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:11.172681 5038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:11.172897 5038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:11.172922 5038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:11.174152 5038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:11.174203 5038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:11.174230 5038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:11.175815 5038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:11.176070 5038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:11.176332 5038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:11.181335 5043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:11.181344 5046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:11.181356 5044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:11.181528 5038 server_base.cc:1034] running on GCE node
I20260923 22:17:11.181766 5038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:11.182017 5038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:11.183158 5038 hybrid_clock.cc:648] HybridClock initialized: now 1790201831183138 us; error 34 us; skew 500 ppm
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:11.185577 5038 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:11.186014 5038 webserver.cc:460] Webserver started at http://127.4.207.1:36073/ using document root <none> and password file <none>
I20260923 22:17:11.186201 5038 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:11.186240 5038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:11.186326 5038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:11.187335 5038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "78f695eb23ee4d638313d9c56e69be2e"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "0f23737db4f3215f30fa5f0f0386e9ce"
server_key_iv: "070b17ac8cafddfaadc888920e5f5b21"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.187729 5038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "78f695eb23ee4d638313d9c56e69be2e"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "0f23737db4f3215f30fa5f0f0386e9ce"
server_key_iv: "070b17ac8cafddfaadc888920e5f5b21"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.189009 5038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:11.189692 5053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.189841 5038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:11.189926 5038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "78f695eb23ee4d638313d9c56e69be2e"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "0f23737db4f3215f30fa5f0f0386e9ce"
server_key_iv: "070b17ac8cafddfaadc888920e5f5b21"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.189985 5038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:11.205870 5038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:11.207958 5038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:11.208071 5038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:11.208245 5038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:11.208590 5038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:11.208623 5038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.208647 5038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:11.208696 5038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.215071 5038 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:36013
I20260923 22:17:11.215150 5166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:36013 every 8 connection(s)
I20260923 22:17:11.215555 5038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:11.222154 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 5038
I20260923 22:17:11.222255 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:11.222427 4924 external_mini_cluster.cc:1428] Setting key 250959579ed90b751ad0752529acc3e4
I20260923 22:17:11.225797 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.216332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:37893 (local address 127.4.207.62:45729)
0923 22:17:11.216376 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:11.216378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:11.216603 (+ 225us) server_negotiation.cc:407] Connection header received
0923 22:17:11.217088 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:11.217089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:11.217101 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:11.217169 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:11.218135 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.218631 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.219675 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.219866 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.219922 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.222042 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:11.222044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:11.222044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:11.222052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:11.224245 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.224785 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.224786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.224787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.224840 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.225147 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.225148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.225148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.225249 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:11.225297 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:11.225656 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 22:17:11.225732 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:17:11.226120 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.216288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:45729 (local address 127.4.207.1:37893)
0923 22:17:11.216453 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:11.216494 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:11.216997 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:11.217226 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:11.217230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:11.217838 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:11.218048 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:11.218053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.218712 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.218712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:11.219600 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:11.219602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.219938 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.219938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:11.220051 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.220599 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:11.220611 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:11.221949 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:11.224332 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:11.224339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:11.224341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:11.224694 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:11.224904 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:11.224905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:11.224906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:11.225079 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:11.225312 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:11.225319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:11.225506 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 22:17:11.225640 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 22:17:11.225829 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260923 22:17:11.226725 5167 heartbeater.cc:347] Connected to a master server at 127.4.207.62:45729
I20260923 22:17:11.226852 5167 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:11.227051 5167 heartbeater.cc:511] Master 127.4.207.62:45729 requested a full tablet report, sending...
I20260923 22:17:11.227705 4976 ts_manager.cc:194] Registered new tserver with Master: 78f695eb23ee4d638313d9c56e69be2e (127.4.207.1:36013)
I20260923 22:17:11.228335 4976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:37893
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:11.250303 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45729
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:11.331483 5172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:11.331737 5172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:11.331775 5172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:11.332857 5172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:11.332930 5172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:11.332975 5172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:11.334447 5172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:11.334719 5172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:11.334955 5172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:11.339861 5178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:11.339944 5180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:11.339911 5177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:11.340394 5172 server_base.cc:1034] running on GCE node
I20260923 22:17:11.340607 5172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:11.340795 5172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:11.341953 5172 hybrid_clock.cc:648] HybridClock initialized: now 1790201831341935 us; error 29 us; skew 500 ppm
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:11.344689 5172 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:11.345182 5172 webserver.cc:460] Webserver started at http://127.4.207.2:39775/ using document root <none> and password file <none>
I20260923 22:17:11.345403 5172 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:11.345467 5172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:11.345618 5172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:11.346658 5172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "334a6dfc6aa4491b8007184c30483e77"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "4fd8b388a57dfbe9841f78b55126c9a7"
server_key_iv: "5fe21be45f47d9322007e3e211a70ea8"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.347025 5172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "334a6dfc6aa4491b8007184c30483e77"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "4fd8b388a57dfbe9841f78b55126c9a7"
server_key_iv: "5fe21be45f47d9322007e3e211a70ea8"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.348469 5172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:11.349206 5187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.349421 5172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:11.349520 5172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "334a6dfc6aa4491b8007184c30483e77"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "4fd8b388a57dfbe9841f78b55126c9a7"
server_key_iv: "5fe21be45f47d9322007e3e211a70ea8"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.349608 5172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:11.365455 5172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:11.367393 5172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:11.367585 5172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:11.367839 5172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:11.368149 5172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:11.368203 5172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.368248 5172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:11.368283 5172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.375072 5172 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:44495
I20260923 22:17:11.375113 5300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:44495 every 8 connection(s)
I20260923 22:17:11.375473 5172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:11.384745 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.376207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:45857 (local address 127.4.207.62:45729)
0923 22:17:11.376292 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:11.376294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:11.376472 (+ 178us) server_negotiation.cc:407] Connection header received
0923 22:17:11.377035 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:11.377037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:11.377049 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:11.377090 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:11.378203 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.378721 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.379847 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.380044 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.380127 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.382149 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:11.382151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:11.382151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:11.382165 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:11.383367 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.383791 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.383793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.383794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.383857 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.384114 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.384114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.384115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.384229 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:11.384302 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:11.384566 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 22:17:11.384659 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:17:11.384927 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.376179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:45729 (local address 127.4.207.2:45857)
0923 22:17:11.376346 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:11.376392 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:11.376901 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:11.377166 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:11.377170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:11.377881 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:11.378084 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:11.378089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.378816 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.378818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:11.379718 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:11.379721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.380134 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.380135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:11.380255 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.380728 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:11.380737 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:11.382041 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:11.383477 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:11.383484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:11.383486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:11.383702 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:11.383930 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:11.383931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:11.383931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:11.384049 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:11.384307 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:11.384313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:11.384442 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 22:17:11.384547 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 22:17:11.384697 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260923 22:17:11.385411 5301 heartbeater.cc:347] Connected to a master server at 127.4.207.62:45729
I20260923 22:17:11.385521 5301 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:11.385515 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 5172
I20260923 22:17:11.385593 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:11.385712 5301 heartbeater.cc:511] Master 127.4.207.62:45729 requested a full tablet report, sending...
I20260923 22:17:11.385761 4924 external_mini_cluster.cc:1428] Setting key 65f299a28f57d1c3ae35529f7b0ce38d
I20260923 22:17:11.386147 4975 ts_manager.cc:194] Registered new tserver with Master: 334a6dfc6aa4491b8007184c30483e77 (127.4.207.2:44495)
I20260923 22:17:11.386571 4975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:45857
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:11.411247 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45729
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:41531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:11.492820 5306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:11.493079 5306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:11.493119 5306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:11.494316 5306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:11.494405 5306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:11.494450 5306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:11.495963 5306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:41531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:11.496251 5306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:11.496482 5306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:11.501641 5306 server_base.cc:1034] running on GCE node
W20260923 22:17:11.501544 5314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:11.501605 5311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:11.501544 5312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:11.502054 5306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:11.502300 5306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:11.503494 5306 hybrid_clock.cc:648] HybridClock initialized: now 1790201831503479 us; error 30 us; skew 500 ppm
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:11.506067 5306 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:11.506521 5306 webserver.cc:460] Webserver started at http://127.4.207.3:45393/ using document root <none> and password file <none>
I20260923 22:17:11.506769 5306 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:11.506840 5306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:11.506990 5306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:11.508136 5306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "9ded8bb08d2d4db4a2c75b5ba2eec428"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "9ec59f3c3a33af3ccb5ca68e45c163e0"
server_key_iv: "124bdd70f9a8e681e0a6ef6c76c31f95"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.508498 5306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "9ded8bb08d2d4db4a2c75b5ba2eec428"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "9ec59f3c3a33af3ccb5ca68e45c163e0"
server_key_iv: "124bdd70f9a8e681e0a6ef6c76c31f95"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.509908 5306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:11.510763 5321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.510991 5306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:11.511094 5306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "9ded8bb08d2d4db4a2c75b5ba2eec428"
format_stamp: "Formatted at 2026-09-23 22:17:11 on dist-test-slave-7511"
server_key: "9ec59f3c3a33af3ccb5ca68e45c163e0"
server_key_iv: "124bdd70f9a8e681e0a6ef6c76c31f95"
server_key_version: "encryptionkey@0"
I20260923 22:17:11.511188 5306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:11.523828 5306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:11.526104 5306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:11.526273 5306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:11.526531 5306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:11.526865 5306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:11.526921 5306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.526973 5306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:11.527004 5306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:11.534438 5306 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:34641
I20260923 22:17:11.534493 5434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:34641 every 8 connection(s)
I20260923 22:17:11.534833 5306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:11.535909 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 5306
I20260923 22:17:11.535975 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:11.536155 4924 external_mini_cluster.cc:1428] Setting key b4efb51610198516e1768ca46feb49ca
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:11.544227 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.535621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:42453 (local address 127.4.207.62:45729)
0923 22:17:11.535688 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:11.535691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:11.535889 (+ 198us) server_negotiation.cc:407] Connection header received
0923 22:17:11.536683 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:11.536684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:11.536696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:11.536750 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:11.537916 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.538424 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.539416 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.539677 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.539732 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.541731 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:11.541732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:11.541732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:11.541739 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:11.542915 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.543288 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.543288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.543289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.543337 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.543607 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.543607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.543608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.543710 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:11.543764 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:11.544094 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 22:17:11.544154 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:17:11.544456 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.535610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:45729 (local address 127.4.207.3:42453)
0923 22:17:11.535762 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:11.535800 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:11.536510 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:11.536846 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:11.536850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:11.537626 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:11.537832 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:11.537837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.538505 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.538506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:11.539342 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:11.539344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.539748 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.539748 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:11.539848 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.540298 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:11.540306 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:11.541647 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:11.543010 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:11.543016 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:11.543018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:11.543223 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:11.543393 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:11.543394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:11.543394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:11.543553 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:11.543841 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:11.543847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:11.543963 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 22:17:11.544069 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 22:17:11.544190 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260923 22:17:11.545118 5435 heartbeater.cc:347] Connected to a master server at 127.4.207.62:45729
I20260923 22:17:11.545228 5435 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:11.545432 5435 heartbeater.cc:511] Master 127.4.207.62:45729 requested a full tablet report, sending...
I20260923 22:17:11.545874 4975 ts_manager.cc:194] Registered new tserver with Master: 9ded8bb08d2d4db4a2c75b5ba2eec428 (127.4.207.3:34641)
I20260923 22:17:11.546296 4975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:42453
I20260923 22:17:11.548277 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:11.572156 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.564352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.4.207.1:36013)
0923 22:17:11.564516 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:11.564526 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:11.564537 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:11.564563 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:11.564565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:11.564579 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:11.564647 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:11.565174 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.565890 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.566597 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.566914 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.566978 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.569255 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:11.569263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:11.569263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:11.569269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:11.570914 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.571266 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.571270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.571270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.571332 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.571602 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.571603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.571604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.571746 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:11.571871 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:11.571916 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 22:17:11.571989 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
W20260923 22:17:11.572523 5081 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39960
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[4931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:11.596249 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:11.588805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.4.207.1:36013)
0923 22:17:11.588877 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:11.588879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:11.588966 (+ 87us) server_negotiation.cc:407] Connection header received
0923 22:17:11.589052 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:11.589053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:11.589064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:11.589125 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:11.589710 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.590218 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.590991 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:11.591191 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:11.591248 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:11.593517 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:11.593519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:11.593519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:11.593526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:11.594968 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.595389 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.595390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.595391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.595591 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:11.595815 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:11.595816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:11.595816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:11.595917 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:11.596015 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:11.596114 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 22:17:11.596179 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 22:17:11.596823 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 5038
I20260923 22:17:11.601249 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 5172
I20260923 22:17:11.606594 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 5306
I20260923 22:17:11.610558 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 4944
2026-09-23T22:17:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4026 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[5449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[5449](info): set up 2 sockets
Sep 23 22:17:11 dist-test-slave-7511 krb5kdc[5449](info): commencing operation
krb5kdc: starting...
W20260923 22:17:12.643043 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:12 dist-test-slave-7511 krb5kdc[5449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:12.762590 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39711
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:39711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:12.842773 5462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:12.843024 5462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:12.843075 5462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:12.844257 5462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:12.844331 5462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:12.844354 5462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:12.844374 5462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:12.844404 5462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:12.845878 5462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:39711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:12.846167 5462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:12.846398 5462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:12.851346 5468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:12.851464 5462 server_base.cc:1034] running on GCE node
W20260923 22:17:12.851337 5470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:12.851341 5467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:12.851825 5462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:12.852177 5462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:12.853356 5462 hybrid_clock.cc:648] HybridClock initialized: now 1790201832853348 us; error 36 us; skew 500 ppm
Sep 23 22:17:12 dist-test-slave-7511 krb5kdc[5449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201832, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:12.855880 5462 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:12.856339 5462 webserver.cc:460] Webserver started at http://127.4.207.62:36249/ using document root <none> and password file <none>
I20260923 22:17:12.856580 5462 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:12.856652 5462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:12.856824 5462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:12.857880 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "a933b9bde9a64760a6f30b2ea5885cad"
format_stamp: "Formatted at 2026-09-23 22:17:12 on dist-test-slave-7511"
server_key: "f9d4d180c88aced050155aeb5499568b"
server_key_iv: "7f6fc6d860931b96363132f02eac9305"
server_key_version: "encryptionkey@0"
I20260923 22:17:12.858184 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "a933b9bde9a64760a6f30b2ea5885cad"
format_stamp: "Formatted at 2026-09-23 22:17:12 on dist-test-slave-7511"
server_key: "f9d4d180c88aced050155aeb5499568b"
server_key_iv: "7f6fc6d860931b96363132f02eac9305"
server_key_version: "encryptionkey@0"
I20260923 22:17:12.859474 5462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:12.860255 5477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:12.860469 5462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:12.860575 5462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "a933b9bde9a64760a6f30b2ea5885cad"
format_stamp: "Formatted at 2026-09-23 22:17:12 on dist-test-slave-7511"
server_key: "f9d4d180c88aced050155aeb5499568b"
server_key_iv: "7f6fc6d860931b96363132f02eac9305"
server_key_version: "encryptionkey@0"
I20260923 22:17:12.860661 5462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:12.875752 5462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:12.877837 5462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:12.878011 5462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:12.882061 5462 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:39711
I20260923 22:17:12.882138 5529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:39711 every 8 connection(s)
I20260923 22:17:12.882493 5462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:12.883430 5530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:12.885927 5530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad: Bootstrap starting.
I20260923 22:17:12.886564 5530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:12.886893 5530 log.cc:824] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:12.887354 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 5462
I20260923 22:17:12.887486 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:12.887687 4924 external_mini_cluster.cc:1428] Setting key d3fefbaae2a0e4fa7a3f70c17eb37ca1
I20260923 22:17:12.887801 5530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad: No bootstrap required, opened a new log
I20260923 22:17:12.889631 5530 raft_consensus.cc:348] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a933b9bde9a64760a6f30b2ea5885cad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39711 } }
I20260923 22:17:12.889765 5530 raft_consensus.cc:374] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:12.889788 5530 raft_consensus.cc:729] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a933b9bde9a64760a6f30b2ea5885cad, State: Initialized, Role: FOLLOWER
I20260923 22:17:12.889922 5530 consensus_queue.cc:260] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a933b9bde9a64760a6f30b2ea5885cad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39711 } }
I20260923 22:17:12.890005 5530 raft_consensus.cc:388] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:12.890048 5530 raft_consensus.cc:482] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:12.890100 5530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:12.890914 5530 raft_consensus.cc:504] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a933b9bde9a64760a6f30b2ea5885cad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39711 } }
I20260923 22:17:12.891058 5530 leader_election.cc:302] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a933b9bde9a64760a6f30b2ea5885cad; no voters:
I20260923 22:17:12.891255 5530 leader_election.cc:288] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:12.891309 5535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:12.891525 5535 raft_consensus.cc:686] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [term 1 LEADER]: Becoming Leader. State: Replica: a933b9bde9a64760a6f30b2ea5885cad, State: Running, Role: LEADER
I20260923 22:17:12.891544 5530 sys_catalog.cc:564] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:12.891681 5535 consensus_queue.cc:237] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a933b9bde9a64760a6f30b2ea5885cad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39711 } }
I20260923 22:17:12.893414 5537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a933b9bde9a64760a6f30b2ea5885cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a933b9bde9a64760a6f30b2ea5885cad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39711 } } }
W20260923 22:17:12.893496 5550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:12.893745 5550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:12.893666 5537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:12.893594 5536 sys_catalog.cc:455] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader a933b9bde9a64760a6f30b2ea5885cad. Latest consensus state: current_term: 1 leader_uuid: "a933b9bde9a64760a6f30b2ea5885cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a933b9bde9a64760a6f30b2ea5885cad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39711 } } }
I20260923 22:17:12.894006 5536 sys_catalog.cc:458] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:12.894012 5552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:12.894299 5552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:12.896097 5552 catalog_manager.cc:1347] Generated new cluster ID: 4b5bfb742afe4d9c8d59516ae24cf968
I20260923 22:17:12.896149 5552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:12 dist-test-slave-7511 krb5kdc[5449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:12.901665 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:12.888302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.4.207.62:39711)
0923 22:17:12.888490 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:12.888501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:12.888542 (+ 41us) server_negotiation.cc:407] Connection header received
0923 22:17:12.889617 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:12.889624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:12.889682 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:12.889850 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:12.890623 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:12.891930 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:12.893139 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:12.893758 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:12.893887 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:12.897152 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:12.897157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:12.897157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:12.897170 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:12.899821 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:12.900251 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:12.900254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:12.900254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:12.900376 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:12.900603 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:12.900604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:12.900604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:12.900896 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:12.901055 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:12.901177 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 22:17:12.901434 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260923 22:17:12.909968 5552 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:12.910507 5552 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:12.913060 5552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a933b9bde9a64760a6f30b2ea5885cad: Generated new TSK 0
I20260923 22:17:12.913245 5552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:12.946193 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39711
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:13.027832 5556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:13.028050 5556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:13.028077 5556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:13.029197 5556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:13.029232 5556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:13.029258 5556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:13.030747 5556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:13.030934 5556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:13.031123 5556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:13.036134 5564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:13.036154 5556 server_base.cc:1034] running on GCE node
W20260923 22:17:13.036145 5561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.036125 5562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:13.036538 5556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:13.036795 5556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:13.037943 5556 hybrid_clock.cc:648] HybridClock initialized: now 1790201833037922 us; error 37 us; skew 500 ppm
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:13.040565 5556 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:13.041004 5556 webserver.cc:460] Webserver started at http://127.4.207.1:34547/ using document root <none> and password file <none>
I20260923 22:17:13.041186 5556 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:13.041227 5556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:13.041314 5556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:13.042294 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "714cb4b54c3d4d8ab44dedd1795b595f"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "953d7e41df5a9e1808ec1cd7cfa5b440"
server_key_iv: "e97fa8997cb3e388464337f0ff161707"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.042599 5556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "714cb4b54c3d4d8ab44dedd1795b595f"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "953d7e41df5a9e1808ec1cd7cfa5b440"
server_key_iv: "e97fa8997cb3e388464337f0ff161707"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.043869 5556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:13.044538 5571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.044705 5556 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:13.044808 5556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "714cb4b54c3d4d8ab44dedd1795b595f"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "953d7e41df5a9e1808ec1cd7cfa5b440"
server_key_iv: "e97fa8997cb3e388464337f0ff161707"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.044883 5556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:13.052444 5556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:13.054324 5556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:13.054481 5556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:13.054706 5556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:13.055007 5556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:13.055063 5556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.055102 5556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:13.055146 5556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.061229 5556 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:46711
I20260923 22:17:13.061275 5684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:46711 every 8 connection(s)
I20260923 22:17:13.061595 5556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:13.070930 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 5556
I20260923 22:17:13.071077 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:13.071287 4924 external_mini_cluster.cc:1428] Setting key bf17546bf570b43222c636fde58f9e6a
I20260923 22:17:13.071470 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.062300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:34751 (local address 127.4.207.62:39711)
0923 22:17:13.062365 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.062367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.062586 (+ 219us) server_negotiation.cc:407] Connection header received
0923 22:17:13.063627 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.063628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.063641 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.063702 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.064302 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.064868 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.065871 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.066082 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.066149 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.068356 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.068357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.068358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.068365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.069890 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.070349 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.070350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.070350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.070406 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.070671 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.070671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.070672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.070819 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.070913 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.071255 (+ 342us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.071353 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 22:17:13.071692 5687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.062273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39711 (local address 127.4.207.1:34751)
0923 22:17:13.062447 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.062497 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.063534 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.063776 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.063780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.064038 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:13.064226 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.064230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.064944 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.064944 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.065770 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.065772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.066173 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.066173 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.066278 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.066753 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:13.066761 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:13.068255 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:13.069979 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.069987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.069990 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.070215 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.070491 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.070492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.070492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.070604 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.070919 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:13.070926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:13.071093 (+ 167us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.071211 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.071392 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260923 22:17:13.072227 5685 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39711
I20260923 22:17:13.072335 5685 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:13.072528 5685 heartbeater.cc:511] Master 127.4.207.62:39711 requested a full tablet report, sending...
I20260923 22:17:13.073052 5494 ts_manager.cc:194] Registered new tserver with Master: 714cb4b54c3d4d8ab44dedd1795b595f (127.4.207.1:46711)
I20260923 22:17:13.073678 5494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:34751
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:13.096203 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39711
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:13.177472 5690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:13.177678 5690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:13.177702 5690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:13.178871 5690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:13.178912 5690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:13.178944 5690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:13.180496 5690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:13.180684 5690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:13.180855 5690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:13.185739 5696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.185803 5695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.185804 5698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:13.186085 5690 server_base.cc:1034] running on GCE node
I20260923 22:17:13.186327 5690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:13.186560 5690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:13.187724 5690 hybrid_clock.cc:648] HybridClock initialized: now 1790201833187705 us; error 39 us; skew 500 ppm
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:13.190204 5690 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:13.190653 5690 webserver.cc:460] Webserver started at http://127.4.207.2:40949/ using document root <none> and password file <none>
I20260923 22:17:13.190850 5690 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:13.190891 5690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:13.190982 5690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:13.192107 5690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "284f7bdfa7cd44f8885b41a8f59073ab"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "febdd89207c737b0e0921c147ea3cb24"
server_key_iv: "6d44b7c82abc1880f5472ab6425091fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.192469 5690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "284f7bdfa7cd44f8885b41a8f59073ab"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "febdd89207c737b0e0921c147ea3cb24"
server_key_iv: "6d44b7c82abc1880f5472ab6425091fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.193773 5690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:13.194410 5705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.194587 5690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:13.194654 5690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "284f7bdfa7cd44f8885b41a8f59073ab"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "febdd89207c737b0e0921c147ea3cb24"
server_key_iv: "6d44b7c82abc1880f5472ab6425091fc"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.194705 5690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:13.213070 5690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:13.215045 5690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:13.215157 5690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:13.215334 5690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:13.215689 5690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:13.215724 5690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.215771 5690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:13.215798 5690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.221572 5690 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:41363
I20260923 22:17:13.221611 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:41363 every 8 connection(s)
I20260923 22:17:13.221915 5690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:13.230988 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 5690
I20260923 22:17:13.231089 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:13.231254 4924 external_mini_cluster.cc:1428] Setting key d497f2b82ded1d9acab8363e5489e10e
I20260923 22:17:13.231897 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.222597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:51657 (local address 127.4.207.62:39711)
0923 22:17:13.222649 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.222652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.222967 (+ 315us) server_negotiation.cc:407] Connection header received
0923 22:17:13.224091 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.224092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.224105 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.224163 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.224834 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.225386 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.226522 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.226746 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.226867 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.229037 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.229039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.229039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.229047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.230316 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.230711 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.230712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.230713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.230768 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.231100 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.231101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.231102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.231232 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.231292 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.231694 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.231802 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:13.231992 5821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.222566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39711 (local address 127.4.207.2:51657)
0923 22:17:13.222757 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.222838 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.223964 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.224250 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.224256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.224538 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:13.224731 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.224735 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.225484 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.225486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.226396 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.226398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.226872 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.226872 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.227011 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.227546 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:13.227555 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:13.228918 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:13.230413 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.230421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.230423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.230641 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.230843 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.230844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.230845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.230995 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.231303 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:13.231311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:13.231516 (+ 205us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.231639 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.231784 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260923 22:17:13.232532 5819 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39711
I20260923 22:17:13.232659 5819 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:13.232847 5819 heartbeater.cc:511] Master 127.4.207.62:39711 requested a full tablet report, sending...
I20260923 22:17:13.233243 5494 ts_manager.cc:194] Registered new tserver with Master: 284f7bdfa7cd44f8885b41a8f59073ab (127.4.207.2:41363)
I20260923 22:17:13.233634 5494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:51657
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:13.255661 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39711
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:13.335682 5824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:13.335908 5824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:13.335945 5824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:13.337098 5824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:13.337164 5824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:13.337209 5824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:13.338693 5824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:13.338953 5824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:13.339178 5824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:13.344110 5832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.344189 5830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:13.344210 5829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:13.344506 5824 server_base.cc:1034] running on GCE node
I20260923 22:17:13.344707 5824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:13.344908 5824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:13.346042 5824 hybrid_clock.cc:648] HybridClock initialized: now 1790201833346030 us; error 29 us; skew 500 ppm
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:13.348564 5824 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:13.349040 5824 webserver.cc:460] Webserver started at http://127.4.207.3:44189/ using document root <none> and password file <none>
I20260923 22:17:13.349259 5824 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:13.349303 5824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:13.349444 5824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:13.350512 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "df73c57012344fd697ad4eeb58a6434d"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "9db5305808f52e8f9e6435bee71b7789"
server_key_iv: "63b60e32ad8ede137c6c39295b2548ba"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.350881 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "df73c57012344fd697ad4eeb58a6434d"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "9db5305808f52e8f9e6435bee71b7789"
server_key_iv: "63b60e32ad8ede137c6c39295b2548ba"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.352316 5824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 22:17:13.353044 5839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.353302 5824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:13.353399 5824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "df73c57012344fd697ad4eeb58a6434d"
format_stamp: "Formatted at 2026-09-23 22:17:13 on dist-test-slave-7511"
server_key: "9db5305808f52e8f9e6435bee71b7789"
server_key_iv: "63b60e32ad8ede137c6c39295b2548ba"
server_key_version: "encryptionkey@0"
I20260923 22:17:13.353475 5824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:13.365705 5824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:13.367668 5824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:13.367820 5824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:13.368043 5824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:13.368353 5824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:13.368412 5824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.368454 5824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:13.368482 5824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:13.374203 5824 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:41201
I20260923 22:17:13.374245 5952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:41201 every 8 connection(s)
I20260923 22:17:13.374547 5824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:13.380240 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 5824
I20260923 22:17:13.380360 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:13.380538 4924 external_mini_cluster.cc:1428] Setting key b79f1a7222df04a5b44e1f94cd315da3
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:13.384135 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.375281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:40281 (local address 127.4.207.62:39711)
0923 22:17:13.375336 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.375339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.375596 (+ 257us) server_negotiation.cc:407] Connection header received
0923 22:17:13.376547 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.376548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.376560 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.376650 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.377267 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.377750 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.378676 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.378895 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.378992 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.381510 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.381511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.381512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.381518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.382964 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.383311 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.383312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.383312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.383362 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.383623 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.383623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.383624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.383727 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.383781 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.384011 (+ 230us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.384071 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:13.384274 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.375254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39711 (local address 127.4.207.3:40281)
0923 22:17:13.375417 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.375493 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.376440 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.376703 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.376707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.376982 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:13.377166 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.377170 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.377829 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.377830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.378599 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.378601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.379007 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.379008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.379117 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.379604 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:13.379612 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:13.381378 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:13.383029 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.383035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.383038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.383244 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.383417 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:13.383418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:13.383418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:13.383575 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:13.383783 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:13.383787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:13.383894 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.383968 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.384085 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260923 22:17:13.384840 5953 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39711
I20260923 22:17:13.384944 5953 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:13.385114 5953 heartbeater.cc:511] Master 127.4.207.62:39711 requested a full tablet report, sending...
I20260923 22:17:13.385507 5494 ts_manager.cc:194] Registered new tserver with Master: df73c57012344fd697ad4eeb58a6434d (127.4.207.3:41201)
I20260923 22:17:13.385840 5494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:40281
I20260923 22:17:13.392457 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:13.400002 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.394246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35638 (local address 127.4.207.62:39711)
0923 22:17:13.394291 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.394293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.394462 (+ 169us) server_negotiation.cc:407] Connection header received
0923 22:17:13.394484 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.394484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.394493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.394537 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.395073 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.395597 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.396286 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.396462 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.396515 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.397613 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.397614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.397614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.397620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.399001 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.399289 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.399290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.399290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.399336 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.399545 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.399546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.399546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.399672 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.399743 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.399828 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.399881 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:13.402137 5494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:13.403818 5494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:13.410415 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.406116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.4.207.2:41363)
0923 22:17:13.406655 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.406659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.406668 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:13.406702 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.406704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.406721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.406799 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:13.407058 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.408218 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.409492 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.409997 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.410095 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.410155 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.410281 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.410326 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":83,"threads_started":1}
I20260923 22:17:13.410452 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.406058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:46711 (local address 127.0.0.1:56338)
0923 22:17:13.406285 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.406322 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.406390 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.406802 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.406803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.406805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:13.406852 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.406854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.407991 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.407992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.409218 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.409221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.410154 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.410155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.410219 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.410223 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.410253 (+ 30us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.410334 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":54,"threads_started":1}
I20260923 22:17:13.410454 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.406058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:41201 (local address 127.0.0.1:45812)
0923 22:17:13.406494 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.406503 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.406550 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.406773 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.406776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.406778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:13.406841 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.406843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.408344 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.408345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.409243 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.409246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.410205 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.410206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.410272 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.410275 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.410320 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.410352 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":2,"spinlock_wait_cycles":384,"thread_start_us":62,"threads_started":1}
I20260923 22:17:13.410463 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.406026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:41363 (local address 127.0.0.1:55316)
0923 22:17:13.406276 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.406320 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.406403 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.406882 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.406883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.406884 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:13.406926 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.406928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.408329 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.408330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.409394 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.409396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.410097 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.410098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.410173 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.410186 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.410256 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.410335 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":63,"thread_start_us":56,"threads_started":1}
I20260923 22:17:13.410748 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.406457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.4.207.1:46711)
0923 22:17:13.406620 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.406624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.406636 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:13.406677 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.406680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.406696 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.406927 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:13.407020 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.407887 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.409308 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.410053 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.410506 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.410577 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.410603 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.410650 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260923 22:17:13.410878 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.406268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.4.207.3:41201)
0923 22:17:13.406458 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.406463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.406542 (+ 79us) server_negotiation.cc:407] Connection header received
0923 22:17:13.406631 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.406633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.406655 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.406744 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:13.406921 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.408217 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.409350 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.410054 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.410124 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.410178 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.410762 (+ 584us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.410804 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260923 22:17:13.411581 5753 tablet_service.cc:1467] Processing CreateTablet for tablet 270682cb10294c6c9d7630bea6173b3d (DEFAULT_TABLE table=test-table [id=d01ed2d7ebf84748bfff87277ea01dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:13.411583 5887 tablet_service.cc:1467] Processing CreateTablet for tablet 270682cb10294c6c9d7630bea6173b3d (DEFAULT_TABLE table=test-table [id=d01ed2d7ebf84748bfff87277ea01dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:13.411574 5619 tablet_service.cc:1467] Processing CreateTablet for tablet 270682cb10294c6c9d7630bea6173b3d (DEFAULT_TABLE table=test-table [id=d01ed2d7ebf84748bfff87277ea01dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:13.411928 5753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270682cb10294c6c9d7630bea6173b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:13.411928 5887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270682cb10294c6c9d7630bea6173b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:13.411954 5619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270682cb10294c6c9d7630bea6173b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:13.415484 5969 tablet_bootstrap.cc:492] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab: Bootstrap starting.
I20260923 22:17:13.415495 5968 tablet_bootstrap.cc:492] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d: Bootstrap starting.
I20260923 22:17:13.415766 5967 tablet_bootstrap.cc:492] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f: Bootstrap starting.
I20260923 22:17:13.416169 5968 tablet_bootstrap.cc:654] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:13.416169 5969 tablet_bootstrap.cc:654] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:13.416340 5967 tablet_bootstrap.cc:654] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:13.416798 5969 log.cc:824] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:13.416797 5968 log.cc:824] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:13.416816 5967 log.cc:824] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:13.417623 5968 tablet_bootstrap.cc:492] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d: No bootstrap required, opened a new log
I20260923 22:17:13.417618 5967 tablet_bootstrap.cc:492] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f: No bootstrap required, opened a new log
I20260923 22:17:13.417689 5967 ts_tablet_manager.cc:1397] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:13.417689 5968 ts_tablet_manager.cc:1397] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:13.419566 5968 raft_consensus.cc:348] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.419842 5969 tablet_bootstrap.cc:492] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab: No bootstrap required, opened a new log
I20260923 22:17:13.419906 5969 ts_tablet_manager.cc:1397] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260923 22:17:13.419566 5967 raft_consensus.cc:348] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.420145 5968 raft_consensus.cc:374] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:13.420184 5967 raft_consensus.cc:374] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:13.420192 5968 raft_consensus.cc:729] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df73c57012344fd697ad4eeb58a6434d, State: Initialized, Role: FOLLOWER
I20260923 22:17:13.420230 5967 raft_consensus.cc:729] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714cb4b54c3d4d8ab44dedd1795b595f, State: Initialized, Role: FOLLOWER
I20260923 22:17:13.420315 5967 consensus_queue.cc:260] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.420311 5968 consensus_queue.cc:260] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.420606 5967 ts_tablet_manager.cc:1428] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:13.420606 5968 ts_tablet_manager.cc:1428] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260923 22:17:13.420625 5685 heartbeater.cc:503] Master 127.4.207.62:39711 was elected leader, sending a full tablet report...
I20260923 22:17:13.420722 5953 heartbeater.cc:503] Master 127.4.207.62:39711 was elected leader, sending a full tablet report...
I20260923 22:17:13.421810 5969 raft_consensus.cc:348] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.422000 5969 raft_consensus.cc:374] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:13.422049 5969 raft_consensus.cc:729] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284f7bdfa7cd44f8885b41a8f59073ab, State: Initialized, Role: FOLLOWER
I20260923 22:17:13.422161 5969 consensus_queue.cc:260] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.422415 5969 ts_tablet_manager.cc:1428] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:17:13.422444 5819 heartbeater.cc:503] Master 127.4.207.62:39711 was elected leader, sending a full tablet report...
W20260923 22:17:13.472524 5820 tablet.cc:2369] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:13.478878 5974 raft_consensus.cc:482] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:13.479025 5974 raft_consensus.cc:504] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.479408 5974 leader_election.cc:288] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 284f7bdfa7cd44f8885b41a8f59073ab (127.4.207.2:41363), 714cb4b54c3d4d8ab44dedd1795b595f (127.4.207.1:46711)
I20260923 22:17:13.483489 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.479725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:58641 (local address 127.4.207.1:46711)
0923 22:17:13.479770 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.479772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.479895 (+ 123us) server_negotiation.cc:407] Connection header received
0923 22:17:13.479975 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.479976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.479993 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.480047 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:13.480246 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.480947 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.482520 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.483103 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.483181 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.483191 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.483364 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.483400 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:13.483510 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.479724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:39565 (local address 127.4.207.2:41363)
0923 22:17:13.479808 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.479811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.479820 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:13.479874 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.479875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.479888 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.479982 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:13.480145 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.480918 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.482518 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.483103 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.483181 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.483191 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.483363 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.483401 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 22:17:13.483515 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.479612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:41363 (local address 127.4.207.3:39565)
0923 22:17:13.479750 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.479762 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.479809 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.479984 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.479985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.479987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:13.480063 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.480065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.481022 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.481023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.482417 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.482420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.483203 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.483204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.483279 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.483285 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.483344 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.483390 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":81}
I20260923 22:17:13.483522 5976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.479612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:46711 (local address 127.4.207.3:58641)
0923 22:17:13.479808 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:13.479818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:13.479908 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:13.480070 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:13.480071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:13.480074 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:13.480142 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.480143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.481022 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.481023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.482417 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:13.482420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.483203 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.483204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:13.483279 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.483286 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 22:17:13.483344 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:17:13.483390 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":128,"thread_start_us":119,"threads_started":1}
I20260923 22:17:13.483839 5773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270682cb10294c6c9d7630bea6173b3d" candidate_uuid: "df73c57012344fd697ad4eeb58a6434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" is_pre_election: true
I20260923 22:17:13.484018 5773 raft_consensus.cc:2454] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df73c57012344fd697ad4eeb58a6434d in term 0.
I20260923 22:17:13.484102 5639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270682cb10294c6c9d7630bea6173b3d" candidate_uuid: "df73c57012344fd697ad4eeb58a6434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" is_pre_election: true
I20260923 22:17:13.484261 5639 raft_consensus.cc:2454] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df73c57012344fd697ad4eeb58a6434d in term 0.
I20260923 22:17:13.484298 5840 leader_election.cc:302] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284f7bdfa7cd44f8885b41a8f59073ab, df73c57012344fd697ad4eeb58a6434d; no voters:
I20260923 22:17:13.484454 5974 raft_consensus.cc:2781] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:13.484514 5974 raft_consensus.cc:482] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:13.484543 5974 raft_consensus.cc:3037] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:13.485237 5974 raft_consensus.cc:504] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.485383 5974 leader_election.cc:288] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [CANDIDATE]: Term 1 election: Requested vote from peers 284f7bdfa7cd44f8885b41a8f59073ab (127.4.207.2:41363), 714cb4b54c3d4d8ab44dedd1795b595f (127.4.207.1:46711)
I20260923 22:17:13.485517 5773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270682cb10294c6c9d7630bea6173b3d" candidate_uuid: "df73c57012344fd697ad4eeb58a6434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284f7bdfa7cd44f8885b41a8f59073ab"
I20260923 22:17:13.485546 5639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270682cb10294c6c9d7630bea6173b3d" candidate_uuid: "df73c57012344fd697ad4eeb58a6434d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714cb4b54c3d4d8ab44dedd1795b595f"
I20260923 22:17:13.485596 5773 raft_consensus.cc:3037] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:13.485605 5639 raft_consensus.cc:3037] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:13.486452 5773 raft_consensus.cc:2454] T 270682cb10294c6c9d7630bea6173b3d P 284f7bdfa7cd44f8885b41a8f59073ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df73c57012344fd697ad4eeb58a6434d in term 1.
I20260923 22:17:13.486452 5639 raft_consensus.cc:2454] T 270682cb10294c6c9d7630bea6173b3d P 714cb4b54c3d4d8ab44dedd1795b595f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df73c57012344fd697ad4eeb58a6434d in term 1.
I20260923 22:17:13.486632 5840 leader_election.cc:302] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 284f7bdfa7cd44f8885b41a8f59073ab, df73c57012344fd697ad4eeb58a6434d; no voters:
I20260923 22:17:13.486750 5974 raft_consensus.cc:2781] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:13.486907 5974 raft_consensus.cc:686] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [term 1 LEADER]: Becoming Leader. State: Replica: df73c57012344fd697ad4eeb58a6434d, State: Running, Role: LEADER
I20260923 22:17:13.487012 5974 consensus_queue.cc:237] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } }
I20260923 22:17:13.487901 5493 catalog_manager.cc:5495] T 270682cb10294c6c9d7630bea6173b3d P df73c57012344fd697ad4eeb58a6434d reported cstate change: term changed from 0 to 1, leader changed from <none> to df73c57012344fd697ad4eeb58a6434d (127.4.207.3). New cstate: current_term: 1 leader_uuid: "df73c57012344fd697ad4eeb58a6434d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284f7bdfa7cd44f8885b41a8f59073ab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 41363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "714cb4b54c3d4d8ab44dedd1795b595f" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df73c57012344fd697ad4eeb58a6434d" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 41201 } health_report { overall_health: HEALTHY } } }
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:13.505992 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.498523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.4.207.1:46711)
0923 22:17:13.498581 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.498583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.498748 (+ 165us) server_negotiation.cc:407] Connection header received
0923 22:17:13.498826 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.498827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.498844 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.498909 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.499479 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.499959 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.500701 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.500894 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.500980 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.503299 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.503310 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.503311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.503318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.504960 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.505294 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.505297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.505297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.505358 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.505526 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.505526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.505527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.505677 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.505793 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.505851 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.505921 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:13.529760 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.522488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56348 (local address 127.4.207.1:46711)
0923 22:17:13.522572 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.522575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.522647 (+ 72us) server_negotiation.cc:407] Connection header received
0923 22:17:13.522749 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.522750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.522760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.522834 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.523421 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.523921 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.524664 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.524873 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.524948 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.527140 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.527141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.527141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.527155 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.528517 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.528891 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.528893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.528893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.528956 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.529156 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.529157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.529157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.529263 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.529363 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.529569 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.529653 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:13.552498 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:13.545205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.4.207.1:46711)
0923 22:17:13.545280 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:13.545282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:13.545339 (+ 57us) server_negotiation.cc:407] Connection header received
0923 22:17:13.545404 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:13.545405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:13.545415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:13.545466 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:13.546042 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.546524 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.547289 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:13.547542 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:13.547617 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:13.549876 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:13.549877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:13.549877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:13.549884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:13.551366 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.551778 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.551779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.551779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.551830 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:13.552038 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:13.552039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:13.552039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:13.552125 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:13.552193 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:13.552331 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 22:17:13.552385 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 22:17:13.553157 5599 tablet_service.cc:2908] Rejecting scan request for tablet 270682cb10294c6c9d7630bea6173b3d: Uninitialized: clean time has not yet been initialized
I20260923 22:17:13.554181 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 5556
I20260923 22:17:13.559352 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 5690
I20260923 22:17:13.563902 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 5824
I20260923 22:17:13.568344 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 5462
2026-09-23T22:17:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1958 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5991](info): set up 2 sockets
Sep 23 22:17:13 dist-test-slave-7511 krb5kdc[5991](info): commencing operation
krb5kdc: starting...
W20260923 22:17:14.611022 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:14 dist-test-slave-7511 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:14.730458 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33149
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:33149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:14.810567 6004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:14.810784 6004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:14.810807 6004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:14.812026 6004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:14.812065 6004 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 22:17:14.812078 6004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:14.812091 6004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:14.812103 6004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:14.812160 6004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:14.813613 6004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:33149
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:14.813870 6004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:14.814117 6004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:14.819123 6012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:14.819159 6010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:14.819190 6004 server_base.cc:1034] running on GCE node
W20260923 22:17:14.819159 6009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:14.819676 6004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:14.819993 6004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:14.821184 6004 hybrid_clock.cc:648] HybridClock initialized: now 1790201834821171 us; error 57 us; skew 500 ppm
Sep 23 22:17:14 dist-test-slave-7511 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:14.823982 6004 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:14.824470 6004 webserver.cc:460] Webserver started at http://127.4.207.62:33429/ using document root <none> and password file <none>
I20260923 22:17:14.824738 6004 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:14.824788 6004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:14.824982 6004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:14.826177 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-7511"
server_key: "979a69169eb643956481dd29dae66605"
server_key_iv: "761524f477e80d263b6e281c5b97aa73"
server_key_version: "encryptionkey@0"
I20260923 22:17:14.826578 6004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-7511"
server_key: "979a69169eb643956481dd29dae66605"
server_key_iv: "761524f477e80d263b6e281c5b97aa73"
server_key_version: "encryptionkey@0"
I20260923 22:17:14.828135 6004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:14.828910 6019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:14.829128 6004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:14.829224 6004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-7511"
server_key: "979a69169eb643956481dd29dae66605"
server_key_iv: "761524f477e80d263b6e281c5b97aa73"
server_key_version: "encryptionkey@0"
I20260923 22:17:14.829280 6004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:14.838711 6004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:14.841025 6004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:14.841138 6004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:14.845537 6004 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:33149
I20260923 22:17:14.845595 6081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:33149 every 8 connection(s)
I20260923 22:17:14.845942 6004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:14.846601 6082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:14.848904 6082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe: Bootstrap starting.
I20260923 22:17:14.849504 6082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:14.849812 6082 log.cc:824] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:14.850461 6082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe: No bootstrap required, opened a new log
I20260923 22:17:14.852581 6082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33149 } }
I20260923 22:17:14.852751 6082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:14.852795 6082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b484ff9d58e4191b9a7e0cfb0eebcbe, State: Initialized, Role: FOLLOWER
I20260923 22:17:14.852974 6082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33149 } }
I20260923 22:17:14.853073 6082 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:14.853116 6082 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:14.853168 6082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:14.853688 6082 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33149 } }
I20260923 22:17:14.853806 6082 leader_election.cc:302] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b484ff9d58e4191b9a7e0cfb0eebcbe; no voters:
I20260923 22:17:14.853996 6082 leader_election.cc:288] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:14.854038 6085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:14.854166 6085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [term 1 LEADER]: Becoming Leader. State: Replica: 0b484ff9d58e4191b9a7e0cfb0eebcbe, State: Running, Role: LEADER
I20260923 22:17:14.854261 6082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:14.854246 6085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33149 } }
I20260923 22:17:14.854633 6086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33149 } } }
I20260923 22:17:14.854645 6087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b484ff9d58e4191b9a7e0cfb0eebcbe. Latest consensus state: current_term: 1 leader_uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b484ff9d58e4191b9a7e0cfb0eebcbe" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33149 } } }
I20260923 22:17:14.854777 6086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:14.854799 6087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:14.854972 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 6004
I20260923 22:17:14.855067 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:14.855091 6091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:14.855234 4924 external_mini_cluster.cc:1428] Setting key bdb0433cb49c69bf4eabf703f0cc4c2f
I20260923 22:17:14.855620 6091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:14.859025 6091 catalog_manager.cc:1347] Generated new cluster ID: 4e789b03df2b4ff7a3ffeb33ef582ba2
I20260923 22:17:14.859079 6091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:14 dist-test-slave-7511 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:14.863881 6091 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:14.864555 6091 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:14.865875 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:14.855715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.4.207.62:33149)
0923 22:17:14.855999 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:14.856008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:14.856038 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:17:14.856227 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:14.856234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:14.856293 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:14.856428 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:14.857206 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:14.858570 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:14.859567 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:14.860000 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:14.860122 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:14.862419 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:14.862422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:14.862423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:14.862434 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:14.864370 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:14.864794 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:14.864797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:14.864797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:14.864887 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:14.865104 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:14.865105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:14.865105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:14.865316 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:14.865495 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:14.865563 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 22:17:14.865711 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":60,"threads_started":1}
I20260923 22:17:14.871176 6091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe: Generated new TSK 0
I20260923 22:17:14.871366 6091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:14.903335 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33149
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:14.988190 6108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:14.988426 6108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:14.988476 6108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:14.989724 6108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:14.989784 6108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:14.989820 6108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:14.989872 6108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:14.991379 6108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:14.991660 6108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:14.991888 6108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:14.996862 6113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:14.996882 6114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:14.996868 6116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:14.997277 6108 server_base.cc:1034] running on GCE node
I20260923 22:17:14.997495 6108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:14.997745 6108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:14.998911 6108 hybrid_clock.cc:648] HybridClock initialized: now 1790201834998892 us; error 33 us; skew 500 ppm
Sep 23 22:17:14 dist-test-slave-7511 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:15.001559 6108 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:15.002022 6108 webserver.cc:460] Webserver started at http://127.4.207.1:44247/ using document root <none> and password file <none>
I20260923 22:17:15.002260 6108 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:15.002328 6108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:15.002470 6108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:15.003641 6108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "49f165352a6f49409f92c7b06bfc02c7"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-7511"
server_key: "e73bc2dae83bc2da1ca6cd2aab61e726"
server_key_iv: "0ca6406d0350afb5482e396231fc8d21"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.004015 6108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "49f165352a6f49409f92c7b06bfc02c7"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-7511"
server_key: "e73bc2dae83bc2da1ca6cd2aab61e726"
server_key_iv: "0ca6406d0350afb5482e396231fc8d21"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.005362 6108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:15.006145 6123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.006309 6108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:15.006412 6108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "49f165352a6f49409f92c7b06bfc02c7"
format_stamp: "Formatted at 2026-09-23 22:17:14 on dist-test-slave-7511"
server_key: "e73bc2dae83bc2da1ca6cd2aab61e726"
server_key_iv: "0ca6406d0350afb5482e396231fc8d21"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.006498 6108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:15.014060 6108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:15.016101 6108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:15.016258 6108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:15.016767 6108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:15.016834 6108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.016901 6108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:15.016929 6108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.023857 6108 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:39709
I20260923 22:17:15.023931 6238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:39709 every 8 connection(s)
Sep 23 22:17:15 dist-test-slave-7511 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:15.024281 6108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:15.027088 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.017335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:38855 (local address 127.4.207.62:33149)
0923 22:17:15.017408 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.017410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.017568 (+ 158us) server_negotiation.cc:407] Connection header received
0923 22:17:15.018049 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.018051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.018064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.018146 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:15.019012 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.019902 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.021512 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.021857 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.021948 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.024748 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:15.024749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:15.024750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:15.024757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:15.025953 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.026320 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.026320 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.026321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.026374 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.026559 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.026560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.026560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.026656 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:15.026707 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.026943 (+ 236us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.027022 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:17:15.027249 6132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.017293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33149 (local address 127.4.207.1:38855)
0923 22:17:15.017442 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.017481 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.017956 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.018242 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.018247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.018600 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:15.018892 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.018899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.020000 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.020001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.021392 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.021395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.021963 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.021963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.022063 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.022640 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:15.022651 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:15.024628 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:15.026025 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.026031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.026032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.026253 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.026415 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.026415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.026416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.026508 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.026713 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:15.026717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:15.026826 (+ 109us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.026910 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.027060 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260923 22:17:15.027833 6239 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33149
I20260923 22:17:15.027915 6239 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:15.028095 6239 heartbeater.cc:511] Master 127.4.207.62:33149 requested a full tablet report, sending...
I20260923 22:17:15.028663 6036 ts_manager.cc:194] Registered new tserver with Master: 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1:39709)
I20260923 22:17:15.028740 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 6108
I20260923 22:17:15.028827 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:15.029022 4924 external_mini_cluster.cc:1428] Setting key cd11e8f0c211e8f0368ce700814bcd0c
I20260923 22:17:15.029461 6036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:38855
I20260923 22:17:15.035548 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.029624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:60109 (local address 127.4.207.62:33149)
0923 22:17:15.029679 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.029681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.029691 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:15.029711 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.029712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.029722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.029773 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:15.030125 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.030778 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.031574 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.031867 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.031937 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.032978 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:15.032979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:15.032980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:15.032986 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:15.034346 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.034729 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.034730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.034730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.034775 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.034994 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.034995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.034996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.035104 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:15.035167 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.035339 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.035397 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:17:15.035614 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.028850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33149 (local address 127.4.207.1:60109)
0923 22:17:15.028978 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.028990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.029040 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.029813 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.029814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.030000 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:15.030050 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.030052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.030900 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.030900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.031425 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.031428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.031955 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.031956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.032042 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.032437 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:15.032442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:15.032878 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:15.034428 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.034429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.034429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.034655 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.034819 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.034820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.034820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.034886 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.035183 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:15.035184 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:15.035236 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.035324 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.035461 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:15.054978 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33149
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:15.136217 6248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:15.136430 6248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:15.136456 6248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:15.137565 6248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:15.137607 6248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:15.137630 6248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:15.137650 6248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:15.139014 6248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:15.139261 6248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:15.139470 6248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:15.144392 6253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:15.144369 6254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:15.144359 6256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:15.144824 6248 server_base.cc:1034] running on GCE node
I20260923 22:17:15.145046 6248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:15.145319 6248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:15.146495 6248 hybrid_clock.cc:648] HybridClock initialized: now 1790201835146473 us; error 35 us; skew 500 ppm
Sep 23 22:17:15 dist-test-slave-7511 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:15.149263 6248 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:15.149720 6248 webserver.cc:460] Webserver started at http://127.4.207.2:37753/ using document root <none> and password file <none>
I20260923 22:17:15.149984 6248 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:15.150058 6248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:15.150213 6248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:15.151327 6248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "a36455e8cfa54473889e3cf7a0dfdb97"
format_stamp: "Formatted at 2026-09-23 22:17:15 on dist-test-slave-7511"
server_key: "a14c635226caeb5351afeeb070276122"
server_key_iv: "ed710ce043c351e84f7ddcf9801845f9"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.151808 6248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "a36455e8cfa54473889e3cf7a0dfdb97"
format_stamp: "Formatted at 2026-09-23 22:17:15 on dist-test-slave-7511"
server_key: "a14c635226caeb5351afeeb070276122"
server_key_iv: "ed710ce043c351e84f7ddcf9801845f9"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.153167 6248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:15.153937 6263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.154124 6248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:15.154220 6248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "a36455e8cfa54473889e3cf7a0dfdb97"
format_stamp: "Formatted at 2026-09-23 22:17:15 on dist-test-slave-7511"
server_key: "a14c635226caeb5351afeeb070276122"
server_key_iv: "ed710ce043c351e84f7ddcf9801845f9"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.154323 6248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:15.198679 6248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:15.202288 6248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:15.202416 6248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:15.203092 6248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:15.203152 6248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.203217 6248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:15.203239 6248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 22:17:15 dist-test-slave-7511 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:15.213280 6248 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:33889
I20260923 22:17:15.213383 6378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:33889 every 8 connection(s)
I20260923 22:17:15.213682 6248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:15.216856 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.203687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:50379 (local address 127.4.207.62:33149)
0923 22:17:15.204719 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.204721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.204733 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:15.204822 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.204823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.204835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.204890 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:15.205711 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.206411 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.208058 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.208381 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.208478 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.211234 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:15.211235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:15.211236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:15.211244 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:15.215194 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.215758 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.215759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.215759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.215830 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.216082 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.216083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.216084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.216241 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:15.216311 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.216680 (+ 369us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.216771 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001}
I20260923 22:17:15.217072 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.203753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33149 (local address 127.4.207.2:50379)
0923 22:17:15.203958 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.204009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.204713 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.205004 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.205008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.205362 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:15.205619 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.205625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.206522 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.206523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.207967 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.207971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.208487 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.208489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.208647 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.209253 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:15.209263 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:15.211100 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:15.215292 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.215298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.215300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.215658 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.215890 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.215891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.215891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.216002 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.216316 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:15.216322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:15.216508 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.216616 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.216812 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":111,"threads_started":1}
I20260923 22:17:15.217623 6379 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33149
I20260923 22:17:15.217716 6379 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:15.217908 6379 heartbeater.cc:511] Master 127.4.207.62:33149 requested a full tablet report, sending...
I20260923 22:17:15.218302 6035 ts_manager.cc:194] Registered new tserver with Master: a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889)
I20260923 22:17:15.218843 6035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:50379
I20260923 22:17:15.220224 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 6248
I20260923 22:17:15.220335 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:15.220526 4924 external_mini_cluster.cc:1428] Setting key 8b6649780ce0c1797b85c49a5a0d4b08
I20260923 22:17:15.225960 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.218706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33149 (local address 127.4.207.2:45405)
0923 22:17:15.218814 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.218825 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.218871 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.219142 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.219143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.219335 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:15.219393 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.219395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.220418 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.220419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.221742 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.221744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.222256 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.222257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.222347 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.222720 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:15.222726 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:15.223120 (+ 394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:15.224783 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.224784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.224785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.225007 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.225234 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.225234 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.225235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.225289 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.225613 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:15.225614 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:15.225677 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.225743 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.225847 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260923 22:17:15.226044 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.218812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:45405 (local address 127.4.207.62:33149)
0923 22:17:15.218855 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.218857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.218867 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:15.218978 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.218979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.218990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.219036 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:15.219555 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.220309 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.221843 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.222141 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.222180 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.223206 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:15.223208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:15.223208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:15.223213 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:15.224676 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.225100 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.225101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.225102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.225159 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.225373 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.225374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.225374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.225556 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:15.225623 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.225796 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.225896 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:15.250052 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33149
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:15.330055 6388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:15.330250 6388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:15.330272 6388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:15.331359 6388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:15.331393 6388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:15.331413 6388 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:15.331430 6388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:15.332947 6388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:33149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:15.333135 6388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:15.333297 6388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:15.338274 6396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:15.338274 6393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:15.338397 6388 server_base.cc:1034] running on GCE node
W20260923 22:17:15.338310 6394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:15.338747 6388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:15.339021 6388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:15.340183 6388 hybrid_clock.cc:648] HybridClock initialized: now 1790201835340172 us; error 44 us; skew 500 ppm
Sep 23 22:17:15 dist-test-slave-7511 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:15.342697 6388 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:15.343223 6388 webserver.cc:460] Webserver started at http://127.4.207.3:37685/ using document root <none> and password file <none>
I20260923 22:17:15.343549 6388 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:15.343633 6388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:15.343786 6388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:15.344837 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "e23126b41a7a43febaebdc0184beafe4"
format_stamp: "Formatted at 2026-09-23 22:17:15 on dist-test-slave-7511"
server_key: "02d548196286f40aac0cf2aa080b879c"
server_key_iv: "b95dcda400bb4f16aae17c0704832185"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.345125 6388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "e23126b41a7a43febaebdc0184beafe4"
format_stamp: "Formatted at 2026-09-23 22:17:15 on dist-test-slave-7511"
server_key: "02d548196286f40aac0cf2aa080b879c"
server_key_iv: "b95dcda400bb4f16aae17c0704832185"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.346328 6388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:15.347021 6403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.347198 6388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:15.347301 6388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "e23126b41a7a43febaebdc0184beafe4"
format_stamp: "Formatted at 2026-09-23 22:17:15 on dist-test-slave-7511"
server_key: "02d548196286f40aac0cf2aa080b879c"
server_key_iv: "b95dcda400bb4f16aae17c0704832185"
server_key_version: "encryptionkey@0"
I20260923 22:17:15.347358 6388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:15.376157 6388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:15.378093 6388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:15.378211 6388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:15.378683 6388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:15.378729 6388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.378757 6388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:15.378803 6388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 22:17:15 dist-test-slave-7511 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:15.386034 6388 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:45825
I20260923 22:17:15.386125 6518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:45825 every 8 connection(s)
I20260923 22:17:15.386440 6388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:15.389320 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.379190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:33391 (local address 127.4.207.62:33149)
0923 22:17:15.379267 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.379270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.379469 (+ 199us) server_negotiation.cc:407] Connection header received
0923 22:17:15.379979 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.379981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.379997 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.380094 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:15.380914 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.381742 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.383311 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.383693 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.383806 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.386558 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:15.386559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:15.386560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:15.386568 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:15.387921 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.388351 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.388353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.388353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.388413 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.388667 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.388668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.388668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.388786 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:15.388856 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.389147 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.389230 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 22:17:15.389521 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.379159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33149 (local address 127.4.207.3:33391)
0923 22:17:15.379297 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.379336 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.379876 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.380177 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.380181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.380524 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:15.380809 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.380814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.381860 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.381861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.383207 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.383211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.383826 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.383827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.383975 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.384573 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:15.384585 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:15.386455 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:15.388018 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.388024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.388027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.388257 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.388478 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.388479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.388480 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.388591 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.388858 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:15.388864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:15.389001 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.389121 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.389276 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260923 22:17:15.390065 6519 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33149
I20260923 22:17:15.390154 6519 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:15.390336 6519 heartbeater.cc:511] Master 127.4.207.62:33149 requested a full tablet report, sending...
I20260923 22:17:15.390738 6036 ts_manager.cc:194] Registered new tserver with Master: e23126b41a7a43febaebdc0184beafe4 (127.4.207.3:45825)
I20260923 22:17:15.391216 6036 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:33391
I20260923 22:17:15.394652 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 6388
I20260923 22:17:15.394769 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:15.394938 4924 external_mini_cluster.cc:1428] Setting key 28ff623348acde208626d8802221adb6
I20260923 22:17:15.395848 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:15.396727 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.391285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:55609 (local address 127.4.207.62:33149)
0923 22:17:15.391328 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.391330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.391538 (+ 208us) server_negotiation.cc:407] Connection header received
0923 22:17:15.391634 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.391635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.391643 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.391691 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:15.392027 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.392480 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.393111 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.393284 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.393335 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.394052 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:15.394053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:15.394054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:15.394060 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:15.395564 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.396004 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.396007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.396007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.396063 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.396250 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.396251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.396251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.396350 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:15.396398 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.396604 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.396669 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:15.396803 6525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.391178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33149 (local address 127.4.207.3:55609)
0923 22:17:15.391485 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.391496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.391568 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.391696 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.391697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.391891 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:15.391955 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.391957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.392560 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.392561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.393041 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.393042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.393346 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.393346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.393396 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.393741 (+ 345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:15.393745 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:15.393995 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:15.395676 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.395677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.395678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.395910 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.396127 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.396128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.396128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.396191 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.396411 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:15.396412 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:15.396468 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.396544 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.396660 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
Sep 23 22:17:15 dist-test-slave-7511 krb5kdc[5991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:15 dist-test-slave-7511 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:15.431329 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.422723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.4.207.62:33149)
0923 22:17:15.422785 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.422804 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.422817 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:17:15.422853 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.422854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.422870 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.422959 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:15.424056 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.424799 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.426032 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.426209 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.426263 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.428920 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:15.428921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:15.428921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:15.428928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:15.430226 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.430592 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.430593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.430593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.430642 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.430793 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.430793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.430794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.430890 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:15.430964 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.431098 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.431261 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:17:15.433147 6036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:15.434711 6036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:15.441260 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.437197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33889 (local address 127.0.0.1:54162)
0923 22:17:15.437436 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.437446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.437499 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.437805 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.437808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.437810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:15.437882 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.437884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.438895 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.438896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.440089 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.440092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.440914 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.440915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.441012 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.441026 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.441124 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.441176 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260923 22:17:15.441267 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.437260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.4.207.2:33889)
0923 22:17:15.437627 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.437632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.437644 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:15.437685 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.437687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.437705 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.437783 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:15.437978 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.438805 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.440184 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.440818 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.440911 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.440970 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.441141 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.441177 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":97,"threads_started":1}
I20260923 22:17:15.441471 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.437181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45825 (local address 127.0.0.1:43966)
0923 22:17:15.437354 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.437397 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.437461 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.438105 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.438106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.438108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:15.438151 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.438152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.439399 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.439400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.440322 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.440324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.441253 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.441254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.441324 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.441329 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.441359 (+ 30us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.441398 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":54,"threads_started":1}
I20260923 22:17:15.441576 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.437368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39709 (local address 127.0.0.1:57890)
0923 22:17:15.437578 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.437586 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.437622 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.437926 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.437928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.437930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:15.437983 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.437984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.439340 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.439341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.440482 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.440485 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.441325 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.441326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.441380 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.441383 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.441410 (+ 27us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.441454 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":34,"thread_start_us":61,"threads_started":1}
I20260923 22:17:15.441747 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.437589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.4.207.1:39709)
0923 22:17:15.437743 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.437747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.437775 (+ 28us) server_negotiation.cc:407] Connection header received
0923 22:17:15.437810 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.437812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.437826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.437907 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:15.438079 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.439251 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.440561 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.441243 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.441513 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.441580 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.441606 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.441648 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260923 22:17:15.441877 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.437294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.4.207.3:45825)
0923 22:17:15.437908 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.437912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.437922 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:15.437990 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.437993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.438008 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.438234 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:15.438294 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.439334 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.440404 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.441145 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.441648 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.441707 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.441730 (+ 23us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.441774 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":82,"threads_started":1}
I20260923 22:17:15.442538 6304 tablet_service.cc:1467] Processing CreateTablet for tablet e5a1bafd79104e74a54bd0d0d31fdfc0 (DEFAULT_TABLE table=test-table [id=c90d9ebe8e2246c18dde53142404a3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:15.442538 6173 tablet_service.cc:1467] Processing CreateTablet for tablet e5a1bafd79104e74a54bd0d0d31fdfc0 (DEFAULT_TABLE table=test-table [id=c90d9ebe8e2246c18dde53142404a3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:15.442903 6304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a1bafd79104e74a54bd0d0d31fdfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.442898 6173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a1bafd79104e74a54bd0d0d31fdfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.443977 6453 tablet_service.cc:1467] Processing CreateTablet for tablet e5a1bafd79104e74a54bd0d0d31fdfc0 (DEFAULT_TABLE table=test-table [id=c90d9ebe8e2246c18dde53142404a3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:15.444285 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5a1bafd79104e74a54bd0d0d31fdfc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.446214 6542 tablet_bootstrap.cc:492] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Bootstrap starting.
I20260923 22:17:15.446580 6543 tablet_bootstrap.cc:492] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Bootstrap starting.
I20260923 22:17:15.447034 6542 tablet_bootstrap.cc:654] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.447227 6543 tablet_bootstrap.cc:654] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.447369 6544 tablet_bootstrap.cc:492] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: Bootstrap starting.
I20260923 22:17:15.447387 6542 log.cc:824] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:15.447739 6543 log.cc:824] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:15.448331 6542 tablet_bootstrap.cc:492] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: No bootstrap required, opened a new log
I20260923 22:17:15.448410 6542 ts_tablet_manager.cc:1397] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:15.448575 6544 tablet_bootstrap.cc:654] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.448596 6543 tablet_bootstrap.cc:492] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: No bootstrap required, opened a new log
I20260923 22:17:15.448706 6543 ts_tablet_manager.cc:1397] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:17:15.448918 6544 log.cc:824] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:15.449645 6544 tablet_bootstrap.cc:492] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: No bootstrap required, opened a new log
I20260923 22:17:15.449728 6544 ts_tablet_manager.cc:1397] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:15.450268 6542 raft_consensus.cc:348] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.450402 6542 raft_consensus.cc:374] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.450450 6542 raft_consensus.cc:729] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f165352a6f49409f92c7b06bfc02c7, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.450588 6542 consensus_queue.cc:260] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.450891 6542 ts_tablet_manager.cc:1428] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:17:15.451025 6239 heartbeater.cc:503] Master 127.4.207.62:33149 was elected leader, sending a full tablet report...
I20260923 22:17:15.451135 6543 raft_consensus.cc:348] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.451285 6543 raft_consensus.cc:374] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.451323 6543 raft_consensus.cc:729] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36455e8cfa54473889e3cf7a0dfdb97, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.451426 6543 consensus_queue.cc:260] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.451742 6543 ts_tablet_manager.cc:1428] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:17:15.451772 6379 heartbeater.cc:503] Master 127.4.207.62:33149 was elected leader, sending a full tablet report...
I20260923 22:17:15.452045 6544 raft_consensus.cc:348] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.452188 6544 raft_consensus.cc:374] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.452207 6544 raft_consensus.cc:729] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23126b41a7a43febaebdc0184beafe4, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.452301 6544 consensus_queue.cc:260] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.452539 6544 ts_tablet_manager.cc:1428] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:15.452646 6519 heartbeater.cc:503] Master 127.4.207.62:33149 was elected leader, sending a full tablet report...
W20260923 22:17:15.464238 6380 tablet.cc:2369] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:15.525060 6240 tablet.cc:2369] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:15.557798 6550 raft_consensus.cc:482] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:15.557922 6550 raft_consensus.cc:504] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.558290 6550 leader_election.cc:288] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889), 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1:39709)
I20260923 22:17:15.560110 6549 raft_consensus.cc:482] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:15.560221 6549 raft_consensus.cc:504] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.560535 6549 leader_election.cc:288] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e23126b41a7a43febaebdc0184beafe4 (127.4.207.3:45825), 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1:39709)
I20260923 22:17:15.561746 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.558433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39709 (local address 127.4.207.3:58393)
0923 22:17:15.558478 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.558505 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.558550 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.558757 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.558758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.558760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:15.558853 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.558855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.559613 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.559614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.560741 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.560744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.561503 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.561504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.561588 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.561591 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.561636 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.561673 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 22:17:15.561909 6538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.558503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:41171 (local address 127.4.207.2:33889)
0923 22:17:15.558550 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.558556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.558854 (+ 298us) server_negotiation.cc:407] Connection header received
0923 22:17:15.558906 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.558907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.558920 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.558973 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:15.559170 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.559763 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.560968 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.561546 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.561585 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.561595 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.561800 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.561845 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:17:15.562000 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.558542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:58393 (local address 127.4.207.1:39709)
0923 22:17:15.558613 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.558615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.558630 (+ 15us) server_negotiation.cc:407] Connection header received
0923 22:17:15.558655 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.558656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.558665 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.558749 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:15.558927 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.559538 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.560828 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.561386 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.561464 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.561474 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.561895 (+ 421us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.561940 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 22:17:15.562165 6551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.558436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33889 (local address 127.4.207.3:41171)
0923 22:17:15.558773 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.558782 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.558853 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.559020 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.559021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.559023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:15.559095 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.559096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.559843 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.559843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.560867 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.560870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.561655 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.561656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.561723 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.561726 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.562011 (+ 285us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.562051 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":171,"thread_start_us":113,"threads_started":1}
I20260923 22:17:15.562247 6193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1bafd79104e74a54bd0d0d31fdfc0" candidate_uuid: "e23126b41a7a43febaebdc0184beafe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f165352a6f49409f92c7b06bfc02c7" is_pre_election: true
I20260923 22:17:15.562387 6193 raft_consensus.cc:2454] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e23126b41a7a43febaebdc0184beafe4 in term 0.
I20260923 22:17:15.562505 6325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1bafd79104e74a54bd0d0d31fdfc0" candidate_uuid: "e23126b41a7a43febaebdc0184beafe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" is_pre_election: true
I20260923 22:17:15.562604 6325 raft_consensus.cc:2454] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e23126b41a7a43febaebdc0184beafe4 in term 0.
I20260923 22:17:15.562610 6404 leader_election.cc:302] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f165352a6f49409f92c7b06bfc02c7, e23126b41a7a43febaebdc0184beafe4; no voters:
I20260923 22:17:15.562726 6550 raft_consensus.cc:2781] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:15.562770 6550 raft_consensus.cc:482] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:15.562795 6550 raft_consensus.cc:3037] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:15.564373 6550 raft_consensus.cc:504] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.564474 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.560680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45825 (local address 127.4.207.2:37521)
0923 22:17:15.560900 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.560909 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.560944 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.561124 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.561125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.561126 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:15.561173 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.561174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.561972 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.561973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.563507 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.563510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.564214 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.564215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.564302 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.564306 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.564369 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.564413 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260923 22:17:15.564473 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.560741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:37521 (local address 127.4.207.3:45825)
0923 22:17:15.561013 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.561014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.561021 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:15.561038 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.561038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.561048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.561092 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:15.561247 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.561875 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.563580 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.564117 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.564188 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.564197 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.564381 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.564416 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260923 22:17:15.564667 6325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1bafd79104e74a54bd0d0d31fdfc0" candidate_uuid: "e23126b41a7a43febaebdc0184beafe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36455e8cfa54473889e3cf7a0dfdb97"
I20260923 22:17:15.564738 6325 raft_consensus.cc:3037] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:15.564723 6473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1bafd79104e74a54bd0d0d31fdfc0" candidate_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23126b41a7a43febaebdc0184beafe4" is_pre_election: true
I20260923 22:17:15.564813 6473 raft_consensus.cc:2379] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a36455e8cfa54473889e3cf7a0dfdb97 in current term 1: Already voted for candidate e23126b41a7a43febaebdc0184beafe4 in this term.
I20260923 22:17:15.565488 6325 raft_consensus.cc:2454] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e23126b41a7a43febaebdc0184beafe4 in term 1.
I20260923 22:17:15.565681 6552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.560765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39709 (local address 127.4.207.2:60851)
0923 22:17:15.561023 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.561033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.561080 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.561455 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.561456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.561457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:15.561533 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.561534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.563141 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.563142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.564802 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.564805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.565422 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.565423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.565488 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.565492 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.565566 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.565604 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20260923 22:17:15.565676 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.560888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:60851 (local address 127.4.207.1:39709)
0923 22:17:15.561316 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.561318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.561324 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:17:15.561344 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.561345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.561353 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.561405 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:15.561609 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.563047 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.564896 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.565324 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.565405 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.565439 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.565574 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.565608 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":68,"threads_started":1}
I20260923 22:17:15.565683 6406 leader_election.cc:302] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a36455e8cfa54473889e3cf7a0dfdb97, e23126b41a7a43febaebdc0184beafe4; no voters:
I20260923 22:17:15.565902 6550 leader_election.cc:288] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [CANDIDATE]: Term 1 election: Requested vote from peers a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889), 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1:39709)
I20260923 22:17:15.565950 6550 raft_consensus.cc:2781] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:15.565994 6550 raft_consensus.cc:686] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 1 LEADER]: Becoming Leader. State: Replica: e23126b41a7a43febaebdc0184beafe4, State: Running, Role: LEADER
I20260923 22:17:15.566080 6193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1bafd79104e74a54bd0d0d31fdfc0" candidate_uuid: "e23126b41a7a43febaebdc0184beafe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f165352a6f49409f92c7b06bfc02c7"
I20260923 22:17:15.566138 6192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5a1bafd79104e74a54bd0d0d31fdfc0" candidate_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f165352a6f49409f92c7b06bfc02c7" is_pre_election: true
I20260923 22:17:15.566200 6550 consensus_queue.cc:237] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } }
I20260923 22:17:15.566155 6193 raft_consensus.cc:3037] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:15.566439 6264 leader_election.cc:302] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a36455e8cfa54473889e3cf7a0dfdb97; no voters: 49f165352a6f49409f92c7b06bfc02c7, e23126b41a7a43febaebdc0184beafe4
I20260923 22:17:15.566581 6549 raft_consensus.cc:2726] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 22:17:15.567139 6193 raft_consensus.cc:2454] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e23126b41a7a43febaebdc0184beafe4 in term 1.
I20260923 22:17:15.567252 6035 catalog_manager.cc:5495] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e23126b41a7a43febaebdc0184beafe4 (127.4.207.3). New cstate: current_term: 1 leader_uuid: "e23126b41a7a43febaebdc0184beafe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:15.586970 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.584314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.4.207.3:45825)
0923 22:17:15.584356 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.584359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.584416 (+ 57us) server_negotiation.cc:407] Connection header received
0923 22:17:15.584574 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.584575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.584599 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.584670 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:15.584818 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.585288 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.586110 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.586313 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.586388 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.586595 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:15.586701 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:15.586794 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.586839 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.586859 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:15.588812 6193 raft_consensus.cc:1261] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Refusing update from remote peer e23126b41a7a43febaebdc0184beafe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:15.588835 6325 raft_consensus.cc:1261] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Refusing update from remote peer e23126b41a7a43febaebdc0184beafe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:15.589154 6554 consensus_queue.cc:1035] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:15.589285 6550 consensus_queue.cc:1035] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:15.592561 6557 mvcc.cc:205] Tried to move back new op lower bound from 7332666718569250816 to 7332666718479822848. Current Snapshot: MvccSnapshot[applied={T|T < 7332666718569250816}]
I20260923 22:17:15.593154 6560 mvcc.cc:205] Tried to move back new op lower bound from 7332666718569250816 to 7332666718479822848. Current Snapshot: MvccSnapshot[applied={T|T < 7332666718569250816}]
I20260923 22:17:15.593571 6558 mvcc.cc:205] Tried to move back new op lower bound from 7332666718569250816 to 7332666718479822848. Current Snapshot: MvccSnapshot[applied={T|T < 7332666718569250816}]
I20260923 22:17:15.596176 6035 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:15.596264 6035 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:15.596967 6035 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe: Sending DeleteTablet for 3 replicas of tablet e5a1bafd79104e74a54bd0d0d31fdfc0
I20260923 22:17:15.597345 6453 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a1bafd79104e74a54bd0d0d31fdfc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:15 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:43966
I20260923 22:17:15.597381 6304 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a1bafd79104e74a54bd0d0d31fdfc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:15 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:54162
I20260923 22:17:15.597606 6566 tablet_replica.cc:330] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: stopping tablet replica
I20260923 22:17:15.597596 6173 tablet_service.cc:1514] Processing DeleteTablet for tablet e5a1bafd79104e74a54bd0d0d31fdfc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:15 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:57890
I20260923 22:17:15.597707 6566 raft_consensus.cc:2229] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:15.597832 6566 raft_consensus.cc:2258] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:15.597791 6035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:15.597935 6035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:15.597838 6568 tablet_replica.cc:330] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: stopping tablet replica
I20260923 22:17:15.598178 6568 raft_consensus.cc:2229] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:15.598304 6566 ts_tablet_manager.cc:1905] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:15.598328 6568 raft_consensus.cc:2258] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:15.598793 6568 ts_tablet_manager.cc:1905] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:15.599264 6453 tablet_service.cc:1467] Processing CreateTablet for tablet 921e42f2500c4e05a212c9fb71fc0743 (DEFAULT_TABLE table=test-table [id=f19dfd6446684ce9adba3056b272d855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:15.599388 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921e42f2500c4e05a212c9fb71fc0743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.599473 6173 tablet_service.cc:1467] Processing CreateTablet for tablet 921e42f2500c4e05a212c9fb71fc0743 (DEFAULT_TABLE table=test-table [id=f19dfd6446684ce9adba3056b272d855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:15.599572 6173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921e42f2500c4e05a212c9fb71fc0743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.600072 6566 ts_tablet_manager.cc:1918] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:15.600145 6566 log.cc:1196] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/e5a1bafd79104e74a54bd0d0d31fdfc0
I20260923 22:17:15.600435 6566 ts_tablet_manager.cc:1939] T e5a1bafd79104e74a54bd0d0d31fdfc0 P e23126b41a7a43febaebdc0184beafe4: Deleting consensus metadata
I20260923 22:17:15.600628 6544 tablet_bootstrap.cc:492] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: Bootstrap starting.
I20260923 22:17:15.600835 6568 ts_tablet_manager.cc:1918] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:15.600953 6542 tablet_bootstrap.cc:492] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: Bootstrap starting.
I20260923 22:17:15.601016 6544 tablet_bootstrap.cc:654] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.600952 6568 log.cc:1196] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/e5a1bafd79104e74a54bd0d0d31fdfc0
I20260923 22:17:15.601363 6568 ts_tablet_manager.cc:1939] T e5a1bafd79104e74a54bd0d0d31fdfc0 P 49f165352a6f49409f92c7b06bfc02c7: Deleting consensus metadata
I20260923 22:17:15.601615 6020 catalog_manager.cc:4841] TS e23126b41a7a43febaebdc0184beafe4 (127.4.207.3:45825): tablet e5a1bafd79104e74a54bd0d0d31fdfc0 (table test-table [id=c90d9ebe8e2246c18dde53142404a3ca]) successfully deleted
I20260923 22:17:15.601660 6542 tablet_bootstrap.cc:654] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.602021 6544 tablet_bootstrap.cc:492] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: No bootstrap required, opened a new log
I20260923 22:17:15.602073 6544 ts_tablet_manager.cc:1397] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:15.602069 6020 catalog_manager.cc:4841] TS 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1:39709): tablet e5a1bafd79104e74a54bd0d0d31fdfc0 (table test-table [id=c90d9ebe8e2246c18dde53142404a3ca]) successfully deleted
I20260923 22:17:15.602178 6304 tablet_service.cc:1467] Processing CreateTablet for tablet 921e42f2500c4e05a212c9fb71fc0743 (DEFAULT_TABLE table=test-table [id=f19dfd6446684ce9adba3056b272d855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:15.602227 6544 raft_consensus.cc:348] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.602288 6304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921e42f2500c4e05a212c9fb71fc0743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.602293 6544 raft_consensus.cc:374] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.602312 6544 raft_consensus.cc:729] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23126b41a7a43febaebdc0184beafe4, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.602339 6542 tablet_bootstrap.cc:492] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: No bootstrap required, opened a new log
I20260923 22:17:15.602353 6544 consensus_queue.cc:260] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.602378 6542 ts_tablet_manager.cc:1397] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:15.602438 6544 ts_tablet_manager.cc:1428] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:15.602515 6542 raft_consensus.cc:348] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.602573 6542 raft_consensus.cc:374] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.602595 6542 raft_consensus.cc:729] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f165352a6f49409f92c7b06bfc02c7, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.602634 6542 consensus_queue.cc:260] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.602711 6542 ts_tablet_manager.cc:1428] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.603657 6543 tablet_bootstrap.cc:492] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: Bootstrap starting.
I20260923 22:17:15.603794 6567 tablet_replica.cc:330] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: stopping tablet replica
I20260923 22:17:15.603895 6567 raft_consensus.cc:2229] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:15.603981 6567 raft_consensus.cc:2258] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:15.604077 6543 tablet_bootstrap.cc:654] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.604337 6567 ts_tablet_manager.cc:1905] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:15.604672 6543 tablet_bootstrap.cc:492] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: No bootstrap required, opened a new log
I20260923 22:17:15.604715 6543 ts_tablet_manager.cc:1397] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:15.604853 6543 raft_consensus.cc:348] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.604910 6543 raft_consensus.cc:374] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.604930 6543 raft_consensus.cc:729] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36455e8cfa54473889e3cf7a0dfdb97, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.604977 6543 consensus_queue.cc:260] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.605059 6543 ts_tablet_manager.cc:1428] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.605703 6567 ts_tablet_manager.cc:1918] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:15.605757 6567 log.cc:1196] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/e5a1bafd79104e74a54bd0d0d31fdfc0
I20260923 22:17:15.605924 6567 ts_tablet_manager.cc:1939] T e5a1bafd79104e74a54bd0d0d31fdfc0 P a36455e8cfa54473889e3cf7a0dfdb97: Deleting consensus metadata
I20260923 22:17:15.606355 6022 catalog_manager.cc:4841] TS a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889): tablet e5a1bafd79104e74a54bd0d0d31fdfc0 (table test-table [id=c90d9ebe8e2246c18dde53142404a3ca]) successfully deleted
W20260923 22:17:15.637032 6520 tablet.cc:2369] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:15.854521 6550 raft_consensus.cc:482] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:15.854615 6550 raft_consensus.cc:504] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.854760 6550 leader_election.cc:288] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1:39709), a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889)
I20260923 22:17:15.854972 6325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921e42f2500c4e05a212c9fb71fc0743" candidate_uuid: "e23126b41a7a43febaebdc0184beafe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" is_pre_election: true
I20260923 22:17:15.854975 6193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921e42f2500c4e05a212c9fb71fc0743" candidate_uuid: "e23126b41a7a43febaebdc0184beafe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f165352a6f49409f92c7b06bfc02c7" is_pre_election: true
I20260923 22:17:15.855072 6325 raft_consensus.cc:2454] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e23126b41a7a43febaebdc0184beafe4 in term 0.
I20260923 22:17:15.855108 6193 raft_consensus.cc:2454] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e23126b41a7a43febaebdc0184beafe4 in term 0.
I20260923 22:17:15.855334 6406 leader_election.cc:302] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a36455e8cfa54473889e3cf7a0dfdb97, e23126b41a7a43febaebdc0184beafe4; no voters:
I20260923 22:17:15.855454 6550 raft_consensus.cc:2781] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:15.855496 6550 raft_consensus.cc:482] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:15.855521 6550 raft_consensus.cc:3037] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:15.856213 6550 raft_consensus.cc:504] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.856372 6550 leader_election.cc:288] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [CANDIDATE]: Term 1 election: Requested vote from peers 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1:39709), a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889)
I20260923 22:17:15.856541 6325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921e42f2500c4e05a212c9fb71fc0743" candidate_uuid: "e23126b41a7a43febaebdc0184beafe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36455e8cfa54473889e3cf7a0dfdb97"
I20260923 22:17:15.856556 6193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921e42f2500c4e05a212c9fb71fc0743" candidate_uuid: "e23126b41a7a43febaebdc0184beafe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f165352a6f49409f92c7b06bfc02c7"
I20260923 22:17:15.856624 6325 raft_consensus.cc:3037] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:15.856650 6193 raft_consensus.cc:3037] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:15.857321 6193 raft_consensus.cc:2454] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e23126b41a7a43febaebdc0184beafe4 in term 1.
I20260923 22:17:15.857311 6325 raft_consensus.cc:2454] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e23126b41a7a43febaebdc0184beafe4 in term 1.
I20260923 22:17:15.857514 6404 leader_election.cc:302] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f165352a6f49409f92c7b06bfc02c7, e23126b41a7a43febaebdc0184beafe4; no voters:
I20260923 22:17:15.857625 6550 raft_consensus.cc:2781] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:15.857672 6550 raft_consensus.cc:686] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 1 LEADER]: Becoming Leader. State: Replica: e23126b41a7a43febaebdc0184beafe4, State: Running, Role: LEADER
I20260923 22:17:15.857729 6550 consensus_queue.cc:237] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.858343 6035 catalog_manager.cc:5495] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e23126b41a7a43febaebdc0184beafe4 (127.4.207.3). New cstate: current_term: 1 leader_uuid: "e23126b41a7a43febaebdc0184beafe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } health_report { overall_health: HEALTHY } } }
Sep 23 22:17:15 dist-test-slave-7511 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:15.911774 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.904273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.4.207.62:33149)
0923 22:17:15.904356 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:15.904358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:15.904429 (+ 71us) server_negotiation.cc:407] Connection header received
0923 22:17:15.904481 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:15.904482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:15.904491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:15.904551 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:15.905005 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.905481 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.906213 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.906387 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.906437 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.908562 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:15.908563 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:15.908564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:15.908570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:15.910429 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.910857 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.910858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.910859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.910929 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:15.911182 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:15.911183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:15.911183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:15.911307 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:15.911370 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:15.911578 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 22:17:15.911666 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 22:17:15.911796 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:15.904206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33149 (local address 127.0.0.1:45996)
0923 22:17:15.904342 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:15.904353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:15.904428 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:15.904573 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:15.904573 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:15.904887 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:15.904939 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.904941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.905558 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.905559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.906141 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:15.906143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:15.906452 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:15.906453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:15.906501 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:15.906993 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:15.906998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:15.908431 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:15.910521 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.910529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.910532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.910755 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.911048 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:15.911049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:15.911050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:15.911111 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:15.911364 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:15.911369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:15.911474 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 22:17:15.911541 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 22:17:15.911724 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":50,"thread_start_us":63,"threads_started":1}
I20260923 22:17:15.913970 6035 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:45996:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 22:17:15.914209 6035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:15.915868 6453 tablet_service.cc:1467] Processing CreateTablet for tablet 0075be4be9ed482796f5888900bcaaef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7d118bf1f524993a46a4f8b245fbc32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:15.915871 6304 tablet_service.cc:1467] Processing CreateTablet for tablet 0075be4be9ed482796f5888900bcaaef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7d118bf1f524993a46a4f8b245fbc32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:15.915871 6173 tablet_service.cc:1467] Processing CreateTablet for tablet 0075be4be9ed482796f5888900bcaaef (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7d118bf1f524993a46a4f8b245fbc32]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:15.916055 6453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0075be4be9ed482796f5888900bcaaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.916055 6173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0075be4be9ed482796f5888900bcaaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.916057 6304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0075be4be9ed482796f5888900bcaaef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:15.917220 6544 tablet_bootstrap.cc:492] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: Bootstrap starting.
I20260923 22:17:15.917881 6542 tablet_bootstrap.cc:492] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: Bootstrap starting.
I20260923 22:17:15.918051 6544 tablet_bootstrap.cc:654] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.918349 6542 tablet_bootstrap.cc:654] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.918671 6544 tablet_bootstrap.cc:492] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: No bootstrap required, opened a new log
I20260923 22:17:15.918705 6543 tablet_bootstrap.cc:492] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: Bootstrap starting.
I20260923 22:17:15.918723 6544 ts_tablet_manager.cc:1397] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:15.918830 6544 raft_consensus.cc:348] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.918917 6544 raft_consensus.cc:374] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.918938 6544 raft_consensus.cc:729] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e23126b41a7a43febaebdc0184beafe4, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.918982 6544 consensus_queue.cc:260] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.919085 6544 ts_tablet_manager.cc:1428] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:15.919088 6543 tablet_bootstrap.cc:654] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:15.919303 6550 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:15.919389 6550 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:15.919580 6542 tablet_bootstrap.cc:492] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: No bootstrap required, opened a new log
I20260923 22:17:15.919636 6542 ts_tablet_manager.cc:1397] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:15.919775 6542 raft_consensus.cc:348] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.919855 6542 raft_consensus.cc:374] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.919880 6542 raft_consensus.cc:729] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f165352a6f49409f92c7b06bfc02c7, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.919926 6542 consensus_queue.cc:260] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.920028 6542 ts_tablet_manager.cc:1428] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.920166 6548 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:15.920272 6548 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:15.920558 6543 tablet_bootstrap.cc:492] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: No bootstrap required, opened a new log
I20260923 22:17:15.920619 6543 ts_tablet_manager.cc:1397] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:17:15.920766 6543 raft_consensus.cc:348] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.920840 6543 raft_consensus.cc:374] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:15.920864 6543 raft_consensus.cc:729] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36455e8cfa54473889e3cf7a0dfdb97, State: Initialized, Role: FOLLOWER
I20260923 22:17:15.920914 6543 consensus_queue.cc:260] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:15.921003 6543 ts_tablet_manager.cc:1428] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:15.921070 6549 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:15.921191 6549 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:16.064484 6548 raft_consensus.cc:482] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:16.064630 6548 raft_consensus.cc:504] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:16.065104 6548 leader_election.cc:288] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889), e23126b41a7a43febaebdc0184beafe4 (127.4.207.3:45825)
I20260923 22:17:16.069582 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.065576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:38203 (local address 127.4.207.3:45825)
0923 22:17:16.065624 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.065626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.065746 (+ 120us) server_negotiation.cc:407] Connection header received
0923 22:17:16.065888 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.065889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.065905 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.065944 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:16.066193 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.067108 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.068666 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.069210 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.069273 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.069284 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.069473 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.069511 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:16.069599 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.065491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:42675 (local address 127.4.207.2:33889)
0923 22:17:16.065678 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.065680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.065772 (+ 92us) server_negotiation.cc:407] Connection header received
0923 22:17:16.065884 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.065886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.065896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.065949 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:16.066137 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.066895 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.068611 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.069189 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.069273 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.069284 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.069473 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.069514 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":81,"threads_started":1}
I20260923 22:17:16.069621 6582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.065389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45825 (local address 127.4.207.1:38203)
0923 22:17:16.065653 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:16.065670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:16.065786 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:16.066029 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:16.066030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:16.066033 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:16.066086 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.066088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.067230 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.067232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.068532 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.068536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.069305 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.069307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.069381 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.069387 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:17:16.069467 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:17:16.069505 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":8,"thread_start_us":237,"threads_started":1}
I20260923 22:17:16.069906 6473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0075be4be9ed482796f5888900bcaaef" candidate_uuid: "49f165352a6f49409f92c7b06bfc02c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23126b41a7a43febaebdc0184beafe4" is_pre_election: true
I20260923 22:17:16.070008 6473 raft_consensus.cc:2454] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f165352a6f49409f92c7b06bfc02c7 in term 0.
I20260923 22:17:16.069617 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.065436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33889 (local address 127.4.207.1:42675)
0923 22:17:16.065686 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:16.065721 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:16.065790 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:16.065996 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:16.065997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:16.065999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:16.066054 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.066056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.066991 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.066992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.068517 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.068520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.069307 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.069308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.069381 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.069386 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:16.069459 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 22:17:16.069505 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":32,"thread_start_us":84,"threads_started":1}
I20260923 22:17:16.070188 6124 leader_election.cc:302] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f165352a6f49409f92c7b06bfc02c7, e23126b41a7a43febaebdc0184beafe4; no voters:
I20260923 22:17:16.070326 6548 raft_consensus.cc:2781] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:16.070363 6548 raft_consensus.cc:482] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:16.070319 6325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0075be4be9ed482796f5888900bcaaef" candidate_uuid: "49f165352a6f49409f92c7b06bfc02c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" is_pre_election: true
I20260923 22:17:16.070389 6548 raft_consensus.cc:3037] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:16.070398 6325 raft_consensus.cc:2454] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49f165352a6f49409f92c7b06bfc02c7 in term 0.
I20260923 22:17:16.071116 6548 raft_consensus.cc:504] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:16.071278 6548 leader_election.cc:288] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [CANDIDATE]: Term 1 election: Requested vote from peers a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889), e23126b41a7a43febaebdc0184beafe4 (127.4.207.3:45825)
I20260923 22:17:16.071463 6473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0075be4be9ed482796f5888900bcaaef" candidate_uuid: "49f165352a6f49409f92c7b06bfc02c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e23126b41a7a43febaebdc0184beafe4"
I20260923 22:17:16.071475 6325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0075be4be9ed482796f5888900bcaaef" candidate_uuid: "49f165352a6f49409f92c7b06bfc02c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a36455e8cfa54473889e3cf7a0dfdb97"
I20260923 22:17:16.071542 6325 raft_consensus.cc:3037] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:16.071542 6473 raft_consensus.cc:3037] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:16.072293 6325 raft_consensus.cc:2454] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f165352a6f49409f92c7b06bfc02c7 in term 1.
I20260923 22:17:16.072325 6473 raft_consensus.cc:2454] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49f165352a6f49409f92c7b06bfc02c7 in term 1.
I20260923 22:17:16.072494 6126 leader_election.cc:302] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49f165352a6f49409f92c7b06bfc02c7, a36455e8cfa54473889e3cf7a0dfdb97; no voters:
I20260923 22:17:16.072592 6548 raft_consensus.cc:2781] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:16.072691 6548 raft_consensus.cc:686] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [term 1 LEADER]: Becoming Leader. State: Replica: 49f165352a6f49409f92c7b06bfc02c7, State: Running, Role: LEADER
I20260923 22:17:16.072789 6548 consensus_queue.cc:237] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } }
I20260923 22:17:16.073465 6585 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: TxnStatusTablet state changed. Reason: New leader 49f165352a6f49409f92c7b06bfc02c7. Latest consensus state: current_term: 1 leader_uuid: "49f165352a6f49409f92c7b06bfc02c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:16.073604 6585 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:16.073765 6587 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 22:17:16.073783 6036 catalog_manager.cc:5495] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1). New cstate: current_term: 1 leader_uuid: "49f165352a6f49409f92c7b06bfc02c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:16.073889 6587 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 22:17:16.103789 6578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.100844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39709 (local address 127.0.0.1:57904)
0923 22:17:16.100965 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:16.100977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:16.101025 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:16.101277 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:16.101278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:16.101279 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:16.101324 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.101326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.102230 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.102230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.102802 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.102803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.103268 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.103269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.103310 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.103324 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:16.103636 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:16.103643 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 22:17:16.103679 (+ 36us) client_negotiation.cc:240] Negotiation successful
0923 22:17:16.103717 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 22:17:16.103820 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.100919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.4.207.1:39709)
0923 22:17:16.101041 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.101043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.101052 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:16.101159 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.101160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.101190 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.101267 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:16.101368 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.102112 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.102885 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.103198 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.103267 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.103414 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:16.103574 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:16.103643 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.103683 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.103705 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260923 22:17:16.105177 6325 raft_consensus.cc:1261] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Refusing update from remote peer 49f165352a6f49409f92c7b06bfc02c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:16.105239 6473 raft_consensus.cc:1261] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Refusing update from remote peer 49f165352a6f49409f92c7b06bfc02c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:16.105506 6585 consensus_queue.cc:1035] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:16.105646 6548 consensus_queue.cc:1035] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:16.106247 6549 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: TxnStatusTablet state changed. Reason: New leader 49f165352a6f49409f92c7b06bfc02c7. Latest consensus state: current_term: 1 leader_uuid: "49f165352a6f49409f92c7b06bfc02c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:16.106256 6550 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: TxnStatusTablet state changed. Reason: New leader 49f165352a6f49409f92c7b06bfc02c7. Latest consensus state: current_term: 1 leader_uuid: "49f165352a6f49409f92c7b06bfc02c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:16.106338 6550 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:16.106334 6549 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:16.107549 6548 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49f165352a6f49409f92c7b06bfc02c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:16.107645 6548 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:16.107708 6548 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49f165352a6f49409f92c7b06bfc02c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:16.107753 6548 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P 49f165352a6f49409f92c7b06bfc02c7: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:16.107980 6550 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49f165352a6f49409f92c7b06bfc02c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:16.108062 6550 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P e23126b41a7a43febaebdc0184beafe4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:16.108500 6549 tablet_replica.cc:439] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49f165352a6f49409f92c7b06bfc02c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 } } peers { permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 } } peers { permanent_uuid: "e23126b41a7a43febaebdc0184beafe4" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45825 } } }
I20260923 22:17:16.108584 6549 tablet_replica.cc:442] T 0075be4be9ed482796f5888900bcaaef P a36455e8cfa54473889e3cf7a0dfdb97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:16.114205 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.111512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:44483 (local address 127.4.207.1:39709)
0923 22:17:16.111554 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.111556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.111694 (+ 138us) server_negotiation.cc:407] Connection header received
0923 22:17:16.111752 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.111752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.111778 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.111819 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:16.111940 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.112437 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.113577 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.113758 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.113815 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.113995 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:16.114020 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:16.114058 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.114135 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.114151 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:16.114267 6602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.111430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39709 (local address 127.4.207.3:44483)
0923 22:17:16.111621 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:16.111632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:16.111694 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:16.111830 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:16.111831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:16.111833 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:16.111884 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.111886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.112512 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.112513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.113483 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.113485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.113833 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.113834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.113895 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.113913 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:16.114079 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:16.114080 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:16.114133 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:17:16.114184 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260923 22:17:16.116585 6325 raft_consensus.cc:1261] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Refusing update from remote peer e23126b41a7a43febaebdc0184beafe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:16.116585 6193 raft_consensus.cc:1261] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Refusing update from remote peer e23126b41a7a43febaebdc0184beafe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:16.116802 6550 consensus_queue.cc:1035] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a36455e8cfa54473889e3cf7a0dfdb97" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:16.116890 6554 consensus_queue.cc:1035] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49f165352a6f49409f92c7b06bfc02c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:16.125684 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.122328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45825 (local address 127.4.207.1:47483)
0923 22:17:16.122470 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:16.122480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:16.122545 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:16.122716 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:16.122716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:16.122725 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:16.122777 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.122778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.124134 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.124135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.124862 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:16.124864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.125324 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.125325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:16.125372 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.125385 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:16.125551 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:16.125553 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:16.125589 (+ 36us) client_negotiation.cc:240] Negotiation successful
0923 22:17:16.125620 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260923 22:17:16.125684 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.122362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:47483 (local address 127.4.207.3:45825)
0923 22:17:16.122404 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.122406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.122555 (+ 149us) server_negotiation.cc:407] Connection header received
0923 22:17:16.122598 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.122599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.122610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.122707 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:16.122834 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.124010 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.124943 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.125249 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.125321 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.125460 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:16.125494 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:16.125547 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.125592 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.125610 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:16.134604 6036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:16.134724 6036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:16.135357 6036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b484ff9d58e4191b9a7e0cfb0eebcbe: Sending DeleteTablet for 3 replicas of tablet 921e42f2500c4e05a212c9fb71fc0743
I20260923 22:17:16.135632 6173 tablet_service.cc:1514] Processing DeleteTablet for tablet 921e42f2500c4e05a212c9fb71fc0743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:16 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:57890
I20260923 22:17:16.135645 6453 tablet_service.cc:1514] Processing DeleteTablet for tablet 921e42f2500c4e05a212c9fb71fc0743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:16 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:43966
I20260923 22:17:16.135771 6304 tablet_service.cc:1514] Processing DeleteTablet for tablet 921e42f2500c4e05a212c9fb71fc0743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:16 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:54162
I20260923 22:17:16.135879 6609 tablet_replica.cc:330] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: stopping tablet replica
I20260923 22:17:16.135939 6609 raft_consensus.cc:2229] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:16.135986 6609 raft_consensus.cc:2258] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:16.136093 6610 tablet_replica.cc:330] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: stopping tablet replica
I20260923 22:17:16.136142 6610 raft_consensus.cc:2229] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:16.136199 6610 raft_consensus.cc:2258] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:16.136427 6609 ts_tablet_manager.cc:1905] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:16.136510 6611 tablet_replica.cc:330] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: stopping tablet replica
I20260923 22:17:16.136576 6611 raft_consensus.cc:2229] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:16.136626 6611 raft_consensus.cc:2258] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:16.136873 6611 ts_tablet_manager.cc:1905] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:16.138324 6609 ts_tablet_manager.cc:1918] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:16.138415 6609 log.cc:1196] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/921e42f2500c4e05a212c9fb71fc0743
I20260923 22:17:16.138578 6611 ts_tablet_manager.cc:1918] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:16.138643 6611 log.cc:1196] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/921e42f2500c4e05a212c9fb71fc0743
I20260923 22:17:16.138643 6609 ts_tablet_manager.cc:1939] T 921e42f2500c4e05a212c9fb71fc0743 P 49f165352a6f49409f92c7b06bfc02c7: Deleting consensus metadata
I20260923 22:17:16.138852 6611 ts_tablet_manager.cc:1939] T 921e42f2500c4e05a212c9fb71fc0743 P a36455e8cfa54473889e3cf7a0dfdb97: Deleting consensus metadata
I20260923 22:17:16.138963 6610 ts_tablet_manager.cc:1905] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:16.139267 6020 catalog_manager.cc:4841] TS 49f165352a6f49409f92c7b06bfc02c7 (127.4.207.1:39709): tablet 921e42f2500c4e05a212c9fb71fc0743 (table test-table [id=f19dfd6446684ce9adba3056b272d855]) successfully deleted
I20260923 22:17:16.139335 6022 catalog_manager.cc:4841] TS a36455e8cfa54473889e3cf7a0dfdb97 (127.4.207.2:33889): tablet 921e42f2500c4e05a212c9fb71fc0743 (table test-table [id=f19dfd6446684ce9adba3056b272d855]) successfully deleted
I20260923 22:17:16.141119 6610 ts_tablet_manager.cc:1918] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:16.141186 6610 log.cc:1196] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/921e42f2500c4e05a212c9fb71fc0743
I20260923 22:17:16.141413 6610 ts_tablet_manager.cc:1939] T 921e42f2500c4e05a212c9fb71fc0743 P e23126b41a7a43febaebdc0184beafe4: Deleting consensus metadata
I20260923 22:17:16.141755 6020 catalog_manager.cc:4841] TS e23126b41a7a43febaebdc0184beafe4 (127.4.207.3:45825): tablet 921e42f2500c4e05a212c9fb71fc0743 (table test-table [id=f19dfd6446684ce9adba3056b272d855]) successfully deleted
Sep 23 22:17:16 dist-test-slave-7511 krb5kdc[5991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201835, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:16.153277 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.136734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.4.207.1:39709)
0923 22:17:16.136772 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.136774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.139098 (+ 2324us) server_negotiation.cc:407] Connection header received
0923 22:17:16.139208 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.139210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.139224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.139254 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:16.140038 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.140775 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.141882 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.142093 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.142125 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.150179 (+ 8054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:16.150199 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:16.150199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:16.150209 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:16.152049 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.152440 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.152451 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.152451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.152508 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.152711 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.152712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.152712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.152858 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:16.153034 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.153079 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.153165 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 22:17:16.153553 6233 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57914
I20260923 22:17:16.161379 6098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:16.154590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.4.207.62:33149)
0923 22:17:16.154636 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:16.154637 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:16.154687 (+ 50us) server_negotiation.cc:407] Connection header received
0923 22:17:16.154753 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:16.154754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:16.154765 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:16.154857 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:16.155409 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.156190 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.156917 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:16.157243 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:16.157320 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:16.158521 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:16.158522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:16.158523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:16.158531 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:16.160233 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.160553 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.160554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.160555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.160621 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:16.160810 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:16.160811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:16.160812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:16.160966 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:16.161119 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:16.161200 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 22:17:16.161280 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 22:17:16.161628 6036 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46002
I20260923 22:17:16.162194 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 6108
I20260923 22:17:16.168308 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 6248
I20260923 22:17:16.173055 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 6388
I20260923 22:17:16.178406 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 6004
2026-09-23T22:17:16Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2608 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:16 dist-test-slave-7511 krb5kdc[6618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:16 dist-test-slave-7511 krb5kdc[6618](info): set up 2 sockets
Sep 23 22:17:16 dist-test-slave-7511 krb5kdc[6618](info): commencing operation
krb5kdc: starting...
W20260923 22:17:17.203130 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:17.318554 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35323
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:17.397971 6631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:17.398168 6631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:17.398190 6631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:17.399331 6631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:17.399374 6631 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 22:17:17.399386 6631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:17.399399 6631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:17.399477 6631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:17.399511 6631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:17.401063 6631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35323
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:17.401305 6631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:17.401553 6631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:17.406615 6637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:17.406653 6636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:17.406642 6631 server_base.cc:1034] running on GCE node
W20260923 22:17:17.406611 6639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:17.407083 6631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:17.407382 6631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:17.408533 6631 hybrid_clock.cc:648] HybridClock initialized: now 1790201837408510 us; error 35 us; skew 500 ppm
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:17.411068 6631 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:17.411597 6631 webserver.cc:460] Webserver started at http://127.4.207.62:38475/ using document root <none> and password file <none>
I20260923 22:17:17.411839 6631 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:17.411918 6631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:17.412055 6631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:17.413151 6631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "9c2d34d28518492888f315de31b22b60"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "a3febb33fcafe618e32a59fc976f8001"
server_key_iv: "14d6f407c701b4f2cb60017bab919d82"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.413511 6631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "9c2d34d28518492888f315de31b22b60"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "a3febb33fcafe618e32a59fc976f8001"
server_key_iv: "14d6f407c701b4f2cb60017bab919d82"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.414794 6631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:17.415557 6646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.415798 6631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:17.415920 6631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "9c2d34d28518492888f315de31b22b60"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "a3febb33fcafe618e32a59fc976f8001"
server_key_iv: "14d6f407c701b4f2cb60017bab919d82"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.416023 6631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:17.429646 6631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:17.431613 6631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:17.431767 6631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:17.436470 6631 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:35323
I20260923 22:17:17.436522 6708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:35323 every 8 connection(s)
I20260923 22:17:17.436828 6631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:17.437449 6709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:17.439680 6709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: Bootstrap starting.
I20260923 22:17:17.440238 6709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:17.440522 6709 log.cc:824] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:17.441205 6709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: No bootstrap required, opened a new log
I20260923 22:17:17.443327 6709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d34d28518492888f315de31b22b60" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35323 } }
I20260923 22:17:17.443495 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 6631
I20260923 22:17:17.443524 6709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:17.443565 6709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2d34d28518492888f315de31b22b60, State: Initialized, Role: FOLLOWER
I20260923 22:17:17.443595 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:17.443715 6709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d34d28518492888f315de31b22b60" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35323 } }
I20260923 22:17:17.443778 6709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:17.443794 4924 external_mini_cluster.cc:1428] Setting key 89d49119d685cc32c90073d6bd45aa2b
I20260923 22:17:17.443833 6709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:17.443881 6709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:17.444556 6709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d34d28518492888f315de31b22b60" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35323 } }
I20260923 22:17:17.444640 6709 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2d34d28518492888f315de31b22b60; no voters:
I20260923 22:17:17.444804 6709 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:17.444842 6714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:17.444990 6714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [term 1 LEADER]: Becoming Leader. State: Replica: 9c2d34d28518492888f315de31b22b60, State: Running, Role: LEADER
I20260923 22:17:17.445078 6709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:17.445078 6714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d34d28518492888f315de31b22b60" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35323 } }
I20260923 22:17:17.445457 6715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2d34d28518492888f315de31b22b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d34d28518492888f315de31b22b60" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35323 } } }
I20260923 22:17:17.445546 6715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:17.445729 6716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2d34d28518492888f315de31b22b60. Latest consensus state: current_term: 1 leader_uuid: "9c2d34d28518492888f315de31b22b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2d34d28518492888f315de31b22b60" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35323 } } }
I20260923 22:17:17.445802 6716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:17.446163 6721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:17.446662 6721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:17.448240 6721 catalog_manager.cc:1347] Generated new cluster ID: 777c73562913432d898e29cdb1a7ba9b
I20260923 22:17:17.448303 6721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:17.455734 6721 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:17.456342 6721 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:17.456648 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.444298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.4.207.62:35323)
0923 22:17:17.444481 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.444490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.444523 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:17:17.444632 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.444639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.444692 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.444825 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.446094 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.447575 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.448672 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.449267 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.449371 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.452172 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.452176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.452177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.452189 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.454895 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.455291 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.455294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.455295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.455446 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.455723 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.455724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.455724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.456004 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.456170 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.456274 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.456429 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260923 22:17:17.469304 6721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: Generated new TSK 0
I20260923 22:17:17.469552 6721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:17.499243 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35323
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:17.595149 6735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:17.595357 6735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:17.595382 6735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:17.596606 6735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:17.596649 6735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:17.596670 6735 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 22:17:17.596683 6735 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:17.596699 6735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:17.598086 6735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:17.598270 6735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:17.598513 6735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:17.604385 6740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:17.604422 6743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:17.604386 6741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:17.604735 6735 server_base.cc:1034] running on GCE node
I20260923 22:17:17.604947 6735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:17.605224 6735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:17.606389 6735 hybrid_clock.cc:648] HybridClock initialized: now 1790201837606382 us; error 39 us; skew 500 ppm
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:17.608955 6735 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:17.609344 6735 webserver.cc:460] Webserver started at http://127.4.207.1:35237/ using document root <none> and password file <none>
I20260923 22:17:17.609575 6735 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:17.609620 6735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:17.609804 6735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:17.610895 6735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "db89ffe0e31e4e009c3cf74633ab0f45"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "f35aec00a0fac129942e61d9f3c9c931"
server_key_iv: "72f7d17b15f59ef6f3cffee96aee2a78"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.611233 6735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "db89ffe0e31e4e009c3cf74633ab0f45"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "f35aec00a0fac129942e61d9f3c9c931"
server_key_iv: "72f7d17b15f59ef6f3cffee96aee2a78"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.612658 6735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:17.613384 6750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.613565 6735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:17.613654 6735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "db89ffe0e31e4e009c3cf74633ab0f45"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "f35aec00a0fac129942e61d9f3c9c931"
server_key_iv: "72f7d17b15f59ef6f3cffee96aee2a78"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.613736 6735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:17.635923 6735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:17.638062 6735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:17.638236 6735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:17.638798 6735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:17.638847 6735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.638873 6735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:17.638924 6735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.645918 6735 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:43731
I20260923 22:17:17.645998 6865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:43731 every 8 connection(s)
I20260923 22:17:17.646283 6735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:17.649288 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.639411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:43597 (local address 127.4.207.62:35323)
0923 22:17:17.639481 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.639483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.639740 (+ 257us) server_negotiation.cc:407] Connection header received
0923 22:17:17.640313 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.640315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.640331 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.640396 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.641391 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.642277 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.643787 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.644119 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.644228 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.646926 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.646927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.646928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.646934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.648179 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.648541 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.648541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.648542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.648591 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.648787 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.648788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.648788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.648885 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.648932 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.649148 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.649227 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:17:17.649451 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.639420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35323 (local address 127.4.207.1:43597)
0923 22:17:17.639621 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.639657 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.640158 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:17.640538 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:17.640542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:17.640945 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:17.641262 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.641268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.642377 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.642378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.643686 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.643688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.644227 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.644227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.644326 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.644903 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:17.644915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:17.646824 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:17.648248 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.648253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.648255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.648473 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.648646 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.648646 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.648647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.648736 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.648938 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:17.648941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:17.649051 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 22:17:17.649128 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 22:17:17.649239 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260923 22:17:17.649993 6866 heartbeater.cc:347] Connected to a master server at 127.4.207.62:35323
I20260923 22:17:17.650079 6866 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:17.650264 6866 heartbeater.cc:511] Master 127.4.207.62:35323 requested a full tablet report, sending...
I20260923 22:17:17.650782 6663 ts_manager.cc:194] Registered new tserver with Master: db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731)
I20260923 22:17:17.651602 6663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:43597
I20260923 22:17:17.655648 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 6735
I20260923 22:17:17.655746 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:17.655923 4924 external_mini_cluster.cc:1428] Setting key d970c62a8ad0eb03be044bf3d9e3e31b
I20260923 22:17:17.656790 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.651123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:53699 (local address 127.4.207.62:35323)
0923 22:17:17.651164 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.651174 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.651239 (+ 65us) server_negotiation.cc:407] Connection header received
0923 22:17:17.651310 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.651311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.651323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.651378 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.651755 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.652430 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.653109 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.653287 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.653336 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.654124 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.654125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.654125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.654131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.655584 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.655983 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.655984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.655985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.656044 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.656248 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.656249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.656249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.656381 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.656455 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.656618 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.656695 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:17.656796 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.651054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35323 (local address 127.4.207.1:53699)
0923 22:17:17.651187 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.651198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.651254 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:17.651383 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:17.651383 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:17.651591 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:17.651637 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.651639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.652523 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.652524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.653042 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.653044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.653347 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.653348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.653397 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.653765 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:17.653768 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:17.654052 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:17.655689 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.655690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.655690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.655896 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.656111 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.656112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.656112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.656174 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.656472 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:17.656473 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:17.656525 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 22:17:17.656592 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 22:17:17.656701 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:17.682426 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35323
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:17.762642 6875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:17.762904 6875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:17.762946 6875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:17.764165 6875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:17.764233 6875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:17.764266 6875 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 22:17:17.764297 6875 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:17.764328 6875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:17.765703 6875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:17.765974 6875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:17.766208 6875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:17.771251 6883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:17.771261 6875 server_base.cc:1034] running on GCE node
W20260923 22:17:17.771226 6881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:17.771207 6880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:17.771708 6875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:17.771984 6875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:17.773151 6875 hybrid_clock.cc:648] HybridClock initialized: now 1790201837773128 us; error 39 us; skew 500 ppm
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:17.775657 6875 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:17.776126 6875 webserver.cc:460] Webserver started at http://127.4.207.2:40461/ using document root <none> and password file <none>
I20260923 22:17:17.776361 6875 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:17.776430 6875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:17.776608 6875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:17.777695 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "4393185ad74e4d41aceebf8743319521"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "7650333c1edbceb1747843aefab3fcff"
server_key_iv: "5948a21e9a82fb7abfae4177a5f4f4bb"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.778093 6875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "4393185ad74e4d41aceebf8743319521"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "7650333c1edbceb1747843aefab3fcff"
server_key_iv: "5948a21e9a82fb7abfae4177a5f4f4bb"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.779606 6875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:17.780468 6890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.780650 6875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:17.780754 6875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "4393185ad74e4d41aceebf8743319521"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "7650333c1edbceb1747843aefab3fcff"
server_key_iv: "5948a21e9a82fb7abfae4177a5f4f4bb"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.780840 6875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:17.790092 6875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:17.792222 6875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:17.792411 6875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:17.792989 6875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:17.793057 6875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.793094 6875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:17.793126 6875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.800326 6875 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:44317
I20260923 22:17:17.800423 7005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:44317 every 8 connection(s)
I20260923 22:17:17.800760 6875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:17.804227 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.793520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:47087 (local address 127.4.207.62:35323)
0923 22:17:17.793568 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.793570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.793759 (+ 189us) server_negotiation.cc:407] Connection header received
0923 22:17:17.794295 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.794296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.794309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.794374 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.795252 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.796101 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.797861 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.798206 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.798298 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.801427 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.801429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.801429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.801436 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.802897 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.803335 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.803336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.803337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.803385 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.803683 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.803684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.803684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.803795 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.803848 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.804097 (+ 249us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.804163 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:17:17.804440 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.793466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35323 (local address 127.4.207.2:47087)
0923 22:17:17.793634 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.793676 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.794188 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:17.794450 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:17.794461 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:17.794805 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:17.795107 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.795114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.796223 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.796225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.797751 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.797755 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.798312 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.798313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.798483 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.799348 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:17.799361 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:17.801327 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:17.803010 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.803018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.803020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.803268 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.803524 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.803524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.803524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.803624 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.803852 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:17.803856 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:17.803982 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 22:17:17.804069 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:17:17.804193 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260923 22:17:17.805058 7006 heartbeater.cc:347] Connected to a master server at 127.4.207.62:35323
I20260923 22:17:17.805181 7006 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:17.805373 7006 heartbeater.cc:511] Master 127.4.207.62:35323 requested a full tablet report, sending...
I20260923 22:17:17.805836 6663 ts_manager.cc:194] Registered new tserver with Master: 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317)
I20260923 22:17:17.806330 6663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:47087
I20260923 22:17:17.807144 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 6875
I20260923 22:17:17.807250 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:17.807428 4924 external_mini_cluster.cc:1428] Setting key 5c7a191634f1e49b5e526984d099d6d5
I20260923 22:17:17.812636 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.806199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:39185 (local address 127.4.207.62:35323)
0923 22:17:17.806243 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.806245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.806367 (+ 122us) server_negotiation.cc:407] Connection header received
0923 22:17:17.806460 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.806461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.806472 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.806536 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.806927 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.807570 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.808575 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.808751 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.808799 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.809896 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.809898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.809898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.809906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.811376 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.811833 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.811834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.811834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.811882 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.812118 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.812119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.812119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.812264 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.812317 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.812516 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.812572 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:17.812772 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.806142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35323 (local address 127.4.207.2:39185)
0923 22:17:17.806289 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.806300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.806383 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:17.806541 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:17.806542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:17.806761 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:17.806842 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.806844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.807678 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.807679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.808483 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.808486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.808821 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.808822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.808905 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.809342 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:17.809347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:17.809813 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:17.811498 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.811499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.811500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.811753 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.811962 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.811963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.811963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.812045 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.812337 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:17.812338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:17.812392 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 22:17:17.812491 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 22:17:17.812615 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:17.833578 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35323
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:17.914695 7015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:17.914955 7015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:17.914994 7015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:17.916141 7015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:17.916213 7015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:17.916265 7015 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 22:17:17.916296 7015 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:17.916328 7015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:17.917871 7015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:35323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:17.918146 7015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:17.918375 7015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:17.923640 7021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:17.923614 7020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:17.923643 7015 server_base.cc:1034] running on GCE node
W20260923 22:17:17.923623 7023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:17.924062 7015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:17.924350 7015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:17.925499 7015 hybrid_clock.cc:648] HybridClock initialized: now 1790201837925488 us; error 31 us; skew 500 ppm
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:17.928114 7015 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:17.928555 7015 webserver.cc:460] Webserver started at http://127.4.207.3:39829/ using document root <none> and password file <none>
I20260923 22:17:17.928783 7015 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:17.928826 7015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:17.928997 7015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:17.930473 7015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "391c7b31a76a4affbc49e2e66daea348"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "02bd30a42a9a8db9cf884f089d646ea5"
server_key_iv: "d53f68d222aaf774cacc3d03c0369dc0"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.930859 7015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "391c7b31a76a4affbc49e2e66daea348"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "02bd30a42a9a8db9cf884f089d646ea5"
server_key_iv: "d53f68d222aaf774cacc3d03c0369dc0"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.932330 7015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260923 22:17:17.933121 7030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.933416 7015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:17.933506 7015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "391c7b31a76a4affbc49e2e66daea348"
format_stamp: "Formatted at 2026-09-23 22:17:17 on dist-test-slave-7511"
server_key: "02bd30a42a9a8db9cf884f089d646ea5"
server_key_iv: "d53f68d222aaf774cacc3d03c0369dc0"
server_key_version: "encryptionkey@0"
I20260923 22:17:17.933578 7015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:17.944676 7015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:17.947604 7015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:17.947772 7015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:17.948348 7015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:17.948387 7015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.948411 7015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:17.948448 7015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:17.955391 7015 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:45601
I20260923 22:17:17.955498 7145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:45601 every 8 connection(s)
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:17.955796 7015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:17.958482 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 7015
I20260923 22:17:17.958591 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:17.958762 4924 external_mini_cluster.cc:1428] Setting key 28971a8e00b0a793e5a26522b74e448f
I20260923 22:17:17.959394 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.948759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:59589 (local address 127.4.207.62:35323)
0923 22:17:17.948829 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.948832 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.949062 (+ 230us) server_negotiation.cc:407] Connection header received
0923 22:17:17.949599 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.949601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.949615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.949685 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.950587 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.951416 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.953069 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.953433 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.953515 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.956297 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.956299 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.956299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.956308 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.957779 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.958202 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.958203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.958204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.958266 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.958520 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.958521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.958521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.958680 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.958738 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.959172 (+ 434us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.959272 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:17.959520 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.948758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35323 (local address 127.4.207.3:59589)
0923 22:17:17.948927 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.948968 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.949451 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:17.949789 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:17.949792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:17.950252 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:17.950470 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.950476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.951551 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.951552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.952964 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.952968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.953526 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.953527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.953625 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.954229 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:17.954240 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:17.956199 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:17.957882 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.957889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.957891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.958122 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.958333 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.958334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.958335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.958449 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.958741 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:17.958748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:17.958930 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 22:17:17.959069 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 22:17:17.959243 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":85,"threads_started":1}
I20260923 22:17:17.959930 7146 heartbeater.cc:347] Connected to a master server at 127.4.207.62:35323
I20260923 22:17:17.960011 7146 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:17.960153 7146 heartbeater.cc:511] Master 127.4.207.62:35323 requested a full tablet report, sending...
I20260923 22:17:17.960489 6662 ts_manager.cc:194] Registered new tserver with Master: 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601)
I20260923 22:17:17.960878 6662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:59589
I20260923 22:17:17.966220 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.960987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35323 (local address 127.4.207.3:44699)
0923 22:17:17.961133 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.961144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.961195 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:17.961315 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:17.961316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:17.961508 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:17.961567 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.961569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.962233 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.962234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.962722 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.962724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.963099 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.963100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:17.963159 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.963672 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:17.963677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:17.963949 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:17.965335 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.965336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.965336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.965503 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.965729 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:17.965729 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:17.965729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:17.965779 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:17.965975 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:17.965976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:17.966034 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 22:17:17.966082 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:17:17.966156 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
I20260923 22:17:17.966225 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.961181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:44699 (local address 127.4.207.62:35323)
0923 22:17:17.961217 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.961218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.961227 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:17.961245 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.961246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.961254 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.961301 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.961644 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.962120 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.962813 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.962986 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.963072 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.964058 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.964059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.964059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.964064 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.965251 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.965606 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.965607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.965608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.965662 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.965833 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.965834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.965834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.965929 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.965986 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.966109 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.966168 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:17:17.970866 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:17 dist-test-slave-7511 krb5kdc[6618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:17.993086 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.986290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.4.207.62:35323)
0923 22:17:17.986357 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.986359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.986368 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:17.986418 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.986418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.986427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.986491 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:17.987036 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.987611 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.988325 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:17.988523 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:17.988572 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:17.990824 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:17.990826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:17.990826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:17.990834 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:17.992171 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.992486 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.992487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.992487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.992535 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:17.992715 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:17.992715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:17.992715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:17.992816 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:17.992887 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:17.992965 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 22:17:17.993022 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:17.994915 6662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:17.996678 6662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:18.003425 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.999100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45601 (local address 127.0.0.1:56028)
0923 22:17:17.999264 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.999310 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.999382 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:17.999688 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:17.999691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:17.999694 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:17.999770 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:17.999772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.000967 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.000968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.002179 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.002181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.003096 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.003097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.003187 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.003205 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.003280 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.003343 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":38,"thread_start_us":73,"threads_started":1}
I20260923 22:17:18.003561 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.999100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:43731 (local address 127.0.0.1:56054)
0923 22:17:17.999763 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.999775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.999834 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.000123 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.000124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.000133 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:18.000180 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.000182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.001152 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.001154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.002365 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.002368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.003284 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.003285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.003361 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.003365 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.003420 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.003484 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":99,"threads_started":1}
I20260923 22:17:18.003599 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.999246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.4.207.1:43731)
0923 22:17:17.999441 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.999446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.999901 (+ 455us) server_negotiation.cc:407] Connection header received
0923 22:17:18.000022 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.000024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.000037 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.000098 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:18.000304 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.001025 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.002465 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.003195 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.003283 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.003350 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.003466 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.003500 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260923 22:17:18.003777 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.999281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.4.207.3:45601)
0923 22:17:17.999516 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:17.999521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:17.999532 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:17.999574 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:17.999577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:17.999594 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:17.999686 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:17.999869 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.000881 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.002275 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.003007 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.003085 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.003145 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.003638 (+ 493us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.003678 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260923 22:17:18.004297 6800 tablet_service.cc:1467] Processing CreateTablet for tablet ff456ad8dc624226b0b213f7d5d27cbe (DEFAULT_TABLE table=test-table [id=c3974c89c21b4806b8d660a2eab5689e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:18.004396 7076 tablet_service.cc:1467] Processing CreateTablet for tablet ff456ad8dc624226b0b213f7d5d27cbe (DEFAULT_TABLE table=test-table [id=c3974c89c21b4806b8d660a2eab5689e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:18.004693 6800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff456ad8dc624226b0b213f7d5d27cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.004693 7076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff456ad8dc624226b0b213f7d5d27cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.005277 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.999305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:44317 (local address 127.0.0.1:41538)
0923 22:17:17.999764 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:17.999775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:17.999833 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.001252 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.001253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.001255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:18.001330 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.001331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.002576 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.002578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.004168 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.004170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.005025 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.005025 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.005069 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.005072 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.005135 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.005208 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":27,"thread_start_us":61,"threads_started":1}
I20260923 22:17:18.005470 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:17.999524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.4.207.2:44317)
0923 22:17:18.001045 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.001050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.001060 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:18.001096 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.001098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.001113 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.001187 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:18.001414 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.002491 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.004245 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.004853 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.004940 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.004996 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.005189 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.005225 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":86,"threads_started":1}
I20260923 22:17:18.008311 7169 tablet_bootstrap.cc:492] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: Bootstrap starting.
I20260923 22:17:18.008280 6940 tablet_service.cc:1467] Processing CreateTablet for tablet ff456ad8dc624226b0b213f7d5d27cbe (DEFAULT_TABLE table=test-table [id=c3974c89c21b4806b8d660a2eab5689e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:18.008498 7170 tablet_bootstrap.cc:492] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: Bootstrap starting.
I20260923 22:17:18.008651 6940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff456ad8dc624226b0b213f7d5d27cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.009063 7170 tablet_bootstrap.cc:654] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.009243 7169 tablet_bootstrap.cc:654] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.009296 7170 log.cc:824] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:18.009570 7169 log.cc:824] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:18.010032 7170 tablet_bootstrap.cc:492] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: No bootstrap required, opened a new log
I20260923 22:17:18.010095 7170 ts_tablet_manager.cc:1397] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:18.010318 7169 tablet_bootstrap.cc:492] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: No bootstrap required, opened a new log
I20260923 22:17:18.010375 7169 ts_tablet_manager.cc:1397] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:18.011731 7170 raft_consensus.cc:348] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.011853 7170 raft_consensus.cc:374] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.011874 7170 raft_consensus.cc:729] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89ffe0e31e4e009c3cf74633ab0f45, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.011994 7170 consensus_queue.cc:260] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.012269 6866 heartbeater.cc:503] Master 127.4.207.62:35323 was elected leader, sending a full tablet report...
I20260923 22:17:18.012599 7174 tablet_bootstrap.cc:492] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Bootstrap starting.
I20260923 22:17:18.012710 7170 ts_tablet_manager.cc:1428] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:18.013212 7174 tablet_bootstrap.cc:654] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.013186 7169 raft_consensus.cc:348] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.013378 7169 raft_consensus.cc:374] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.013413 7169 raft_consensus.cc:729] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391c7b31a76a4affbc49e2e66daea348, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.013548 7169 consensus_queue.cc:260] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.013569 7174 log.cc:824] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:18.013809 7169 ts_tablet_manager.cc:1428] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:18.013835 7146 heartbeater.cc:503] Master 127.4.207.62:35323 was elected leader, sending a full tablet report...
I20260923 22:17:18.014654 7174 tablet_bootstrap.cc:492] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: No bootstrap required, opened a new log
I20260923 22:17:18.014724 7174 ts_tablet_manager.cc:1397] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:18.016155 7174 raft_consensus.cc:348] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.016304 7174 raft_consensus.cc:374] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.016350 7174 raft_consensus.cc:729] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4393185ad74e4d41aceebf8743319521, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.016469 7174 consensus_queue.cc:260] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.016736 7174 ts_tablet_manager.cc:1428] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:18.016837 7006 heartbeater.cc:503] Master 127.4.207.62:35323 was elected leader, sending a full tablet report...
I20260923 22:17:18.018348 7173 raft_consensus.cc:482] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:18.018428 7173 raft_consensus.cc:504] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.018797 7173 leader_election.cc:288] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317), 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601)
I20260923 22:17:18.021878 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.019117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:57743 (local address 127.4.207.2:44317)
0923 22:17:18.019179 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.019181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.019188 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:18.019217 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.019217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.019225 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.019297 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:18.019480 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.020111 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.021131 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.021533 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.021589 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.021598 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.021765 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.021812 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:17:18.021881 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.018987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:44317 (local address 127.4.207.1:57743)
0923 22:17:18.019103 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.019113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.019151 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.019303 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.019304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.019306 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:18.019378 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.019379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.020190 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.020192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.021007 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.021009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.021613 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.021614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.021692 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.021695 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.021746 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.021785 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 22:17:18.021940 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.019102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:33417 (local address 127.4.207.3:45601)
0923 22:17:18.019147 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.019149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.019413 (+ 264us) server_negotiation.cc:407] Connection header received
0923 22:17:18.019470 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.019471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.019489 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.019535 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:18.019734 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.020228 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.021211 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.021560 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.021662 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.021673 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.021829 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.021865 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:18.021991 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.018987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45601 (local address 127.4.207.1:33417)
0923 22:17:18.019340 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.019350 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.019406 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.019575 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.019575 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.019577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:18.019635 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.019637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.020321 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.020323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.021124 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.021128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.021666 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.021667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.021754 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.021759 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.021814 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.021850 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":200,"thread_start_us":108,"threads_started":1}
I20260923 22:17:18.022282 6960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff456ad8dc624226b0b213f7d5d27cbe" candidate_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393185ad74e4d41aceebf8743319521" is_pre_election: true
I20260923 22:17:18.022332 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff456ad8dc624226b0b213f7d5d27cbe" candidate_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391c7b31a76a4affbc49e2e66daea348" is_pre_election: true
I20260923 22:17:18.022457 7100 raft_consensus.cc:2454] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89ffe0e31e4e009c3cf74633ab0f45 in term 0.
I20260923 22:17:18.022457 6960 raft_consensus.cc:2454] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89ffe0e31e4e009c3cf74633ab0f45 in term 0.
I20260923 22:17:18.022689 6754 leader_election.cc:302] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391c7b31a76a4affbc49e2e66daea348, db89ffe0e31e4e009c3cf74633ab0f45; no voters:
I20260923 22:17:18.022830 7173 raft_consensus.cc:2781] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:18.022884 7173 raft_consensus.cc:482] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:18.022940 7173 raft_consensus.cc:3037] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.023746 7173 raft_consensus.cc:504] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.023910 7173 leader_election.cc:288] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [CANDIDATE]: Term 1 election: Requested vote from peers 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317), 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601)
I20260923 22:17:18.024143 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff456ad8dc624226b0b213f7d5d27cbe" candidate_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391c7b31a76a4affbc49e2e66daea348"
I20260923 22:17:18.024158 6960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff456ad8dc624226b0b213f7d5d27cbe" candidate_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393185ad74e4d41aceebf8743319521"
I20260923 22:17:18.024224 6960 raft_consensus.cc:3037] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.024225 7100 raft_consensus.cc:3037] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.025077 6960 raft_consensus.cc:2454] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89ffe0e31e4e009c3cf74633ab0f45 in term 1.
I20260923 22:17:18.025077 7100 raft_consensus.cc:2454] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89ffe0e31e4e009c3cf74633ab0f45 in term 1.
I20260923 22:17:18.025278 6754 leader_election.cc:302] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391c7b31a76a4affbc49e2e66daea348, db89ffe0e31e4e009c3cf74633ab0f45; no voters:
I20260923 22:17:18.025357 7173 raft_consensus.cc:2781] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:18.025535 7173 raft_consensus.cc:686] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 LEADER]: Becoming Leader. State: Replica: db89ffe0e31e4e009c3cf74633ab0f45, State: Running, Role: LEADER
I20260923 22:17:18.025648 7173 consensus_queue.cc:237] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.026520 6662 catalog_manager.cc:5495] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 reported cstate change: term changed from 0 to 1, leader changed from <none> to db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1). New cstate: current_term: 1 leader_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } health_report { overall_health: UNKNOWN } } }
W20260923 22:17:18.051590 7007 tablet.cc:2369] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 22:17:18 dist-test-slave-7511 krb5kdc[6618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:18.064600 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.056364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.4.207.62:35323)
0923 22:17:18.056430 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.056432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.056558 (+ 126us) server_negotiation.cc:407] Connection header received
0923 22:17:18.056594 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.056595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.056605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.056672 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:18.057150 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.057688 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.058463 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.058775 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.058863 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.061716 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:18.061717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:18.061717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:18.061725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:18.063426 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.063875 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.063876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.063877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.063923 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:18.064084 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:18.064085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:18.064085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:18.064200 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:18.064248 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.064421 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.064506 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:17:18.064865 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.056266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35323 (local address 127.0.0.1:51536)
0923 22:17:18.056479 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.056489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.056552 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.056692 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.056693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.057025 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:18.057081 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.057083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.057780 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.057781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.058352 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.058354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.058863 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.058864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.058958 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.059672 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:18.059677 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:18.061580 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:18.063539 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:18.063547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:18.063551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:18.063800 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:18.063980 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:18.063981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:18.063981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:18.064029 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:18.064283 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:18.064289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:18.064329 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.064696 (+ 367us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.064788 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":112,"threads_started":1}
I20260923 22:17:18.070468 6662 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:51536:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 22:17:18.070669 6662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:18.072332 7076 tablet_service.cc:1467] Processing CreateTablet for tablet ab7a1ba38f244ddb9040f55fa9f2396e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3017a866b804da9af54e90633a65f3d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:18.072522 7076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7a1ba38f244ddb9040f55fa9f2396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.072484 6800 tablet_service.cc:1467] Processing CreateTablet for tablet ab7a1ba38f244ddb9040f55fa9f2396e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3017a866b804da9af54e90633a65f3d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:18.072664 6800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7a1ba38f244ddb9040f55fa9f2396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.073928 7170 tablet_bootstrap.cc:492] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: Bootstrap starting.
I20260923 22:17:18.074018 7169 tablet_bootstrap.cc:492] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: Bootstrap starting.
I20260923 22:17:18.074038 6940 tablet_service.cc:1467] Processing CreateTablet for tablet ab7a1ba38f244ddb9040f55fa9f2396e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3017a866b804da9af54e90633a65f3d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:18.074180 6940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab7a1ba38f244ddb9040f55fa9f2396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.074419 7170 tablet_bootstrap.cc:654] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.074579 7169 tablet_bootstrap.cc:654] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.075181 7170 tablet_bootstrap.cc:492] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: No bootstrap required, opened a new log
I20260923 22:17:18.075212 7169 tablet_bootstrap.cc:492] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: No bootstrap required, opened a new log
I20260923 22:17:18.075232 7170 ts_tablet_manager.cc:1397] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:18.075248 7169 ts_tablet_manager.cc:1397] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:18.075382 7170 raft_consensus.cc:348] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.075382 7169 raft_consensus.cc:348] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.075476 7169 raft_consensus.cc:374] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.075476 7170 raft_consensus.cc:374] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.075500 7170 raft_consensus.cc:729] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89ffe0e31e4e009c3cf74633ab0f45, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.075500 7169 raft_consensus.cc:729] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391c7b31a76a4affbc49e2e66daea348, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.075549 7169 consensus_queue.cc:260] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.075544 7170 consensus_queue.cc:260] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.075634 7169 ts_tablet_manager.cc:1428] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.075646 7170 ts_tablet_manager.cc:1428] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.075687 7173 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.075778 7173 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:18.075918 7175 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.076016 7175 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:18.076229 7174 tablet_bootstrap.cc:492] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: Bootstrap starting.
I20260923 22:17:18.076915 7174 tablet_bootstrap.cc:654] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.077522 7174 tablet_bootstrap.cc:492] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: No bootstrap required, opened a new log
I20260923 22:17:18.077593 7174 ts_tablet_manager.cc:1397] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:18.077745 7174 raft_consensus.cc:348] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.077852 7174 raft_consensus.cc:374] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.077885 7174 raft_consensus.cc:729] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4393185ad74e4d41aceebf8743319521, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.077942 7174 consensus_queue.cc:260] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.078035 7174 ts_tablet_manager.cc:1428] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.078117 7177 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.078254 7177 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 22:17:18.147109 6867 tablet.cc:2369] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:18.157963 7175 raft_consensus.cc:482] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:18.158067 7175 raft_consensus.cc:504] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.158442 7175 leader_election.cc:288] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731), 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317)
I20260923 22:17:18.161684 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.158691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:43731 (local address 127.4.207.3:48473)
0923 22:17:18.158760 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.158774 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.158841 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.159024 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.159025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.159026 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:18.159098 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.159100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.159866 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.159867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.160857 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.160859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.161414 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.161415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.161498 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.161502 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.161555 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.161586 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 22:17:18.161716 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.158781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:48473 (local address 127.4.207.1:43731)
0923 22:17:18.158826 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.158828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.158839 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:18.158919 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.158921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.158934 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.158996 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:18.159182 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.159793 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.160920 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.161331 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.161386 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.161394 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.161576 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.161626 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:17:18.161765 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.158903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:59843 (local address 127.4.207.2:44317)
0923 22:17:18.158945 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.158948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.159070 (+ 122us) server_negotiation.cc:407] Connection header received
0923 22:17:18.159171 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.159172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.159181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.159236 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:18.159407 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.160051 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.160998 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.161382 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.161453 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.161463 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.161652 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.161694 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:18.161974 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.158833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:44317 (local address 127.4.207.3:59843)
0923 22:17:18.158988 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.158998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.159081 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.159250 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.159251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.159252 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:18.159327 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.159328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.160129 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.160130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.160905 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.160908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.161493 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.161494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.161566 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.161573 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.161629 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.161667 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260923 22:17:18.162259 6960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7a1ba38f244ddb9040f55fa9f2396e" candidate_uuid: "391c7b31a76a4affbc49e2e66daea348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393185ad74e4d41aceebf8743319521" is_pre_election: true
I20260923 22:17:18.162377 6960 raft_consensus.cc:2454] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391c7b31a76a4affbc49e2e66daea348 in term 0.
I20260923 22:17:18.162358 6820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7a1ba38f244ddb9040f55fa9f2396e" candidate_uuid: "391c7b31a76a4affbc49e2e66daea348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" is_pre_election: true
I20260923 22:17:18.162492 6820 raft_consensus.cc:2454] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391c7b31a76a4affbc49e2e66daea348 in term 0.
I20260923 22:17:18.162611 7031 leader_election.cc:302] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391c7b31a76a4affbc49e2e66daea348, 4393185ad74e4d41aceebf8743319521; no voters:
I20260923 22:17:18.162784 7175 raft_consensus.cc:2781] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:18.162828 7175 raft_consensus.cc:482] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:18.162849 7175 raft_consensus.cc:3037] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.163609 7175 raft_consensus.cc:504] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.163741 7175 leader_election.cc:288] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [CANDIDATE]: Term 1 election: Requested vote from peers db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731), 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317)
I20260923 22:17:18.163884 6820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7a1ba38f244ddb9040f55fa9f2396e" candidate_uuid: "391c7b31a76a4affbc49e2e66daea348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89ffe0e31e4e009c3cf74633ab0f45"
I20260923 22:17:18.163976 6820 raft_consensus.cc:3037] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.163928 6960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab7a1ba38f244ddb9040f55fa9f2396e" candidate_uuid: "391c7b31a76a4affbc49e2e66daea348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393185ad74e4d41aceebf8743319521"
I20260923 22:17:18.164021 6960 raft_consensus.cc:3037] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.164614 6820 raft_consensus.cc:2454] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391c7b31a76a4affbc49e2e66daea348 in term 1.
I20260923 22:17:18.164620 6960 raft_consensus.cc:2454] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391c7b31a76a4affbc49e2e66daea348 in term 1.
I20260923 22:17:18.164778 7031 leader_election.cc:302] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391c7b31a76a4affbc49e2e66daea348, 4393185ad74e4d41aceebf8743319521; no voters:
I20260923 22:17:18.164908 7175 raft_consensus.cc:2781] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:18.165100 7175 raft_consensus.cc:686] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [term 1 LEADER]: Becoming Leader. State: Replica: 391c7b31a76a4affbc49e2e66daea348, State: Running, Role: LEADER
I20260923 22:17:18.165217 7175 consensus_queue.cc:237] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.165717 7191 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: TxnStatusTablet state changed. Reason: New leader 391c7b31a76a4affbc49e2e66daea348. Latest consensus state: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.165804 7191 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:18.165997 7193 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 22:17:18.166110 7193 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 22:17:18.166137 6662 catalog_manager.cc:5495] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 reported cstate change: term changed from 0 to 1, leader changed from <none> to 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3). New cstate: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } health_report { overall_health: HEALTHY } } }
I20260923 22:17:18.172991 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.170497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.4.207.3:45601)
0923 22:17:18.170555 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.170557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.170588 (+ 31us) server_negotiation.cc:407] Connection header received
0923 22:17:18.170675 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.170676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.170692 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.170745 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:18.170892 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.171367 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.172173 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.172383 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.172454 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.172650 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.172748 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.172833 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.172882 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.172901 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:17:18.173017 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.170443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45601 (local address 127.0.0.1:56032)
0923 22:17:18.170543 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.170554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.170608 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.170758 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.170759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.170760 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:18.170811 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.170813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.171478 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.171479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.172067 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.172068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.172465 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.172466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.172534 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.172548 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.172812 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.172822 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.172878 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.172945 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 22:17:18.174801 6960 raft_consensus.cc:1261] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Refusing update from remote peer 391c7b31a76a4affbc49e2e66daea348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:18.174847 6820 raft_consensus.cc:1261] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Refusing update from remote peer 391c7b31a76a4affbc49e2e66daea348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:18.175088 7175 consensus_queue.cc:1035] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:18.175228 7191 consensus_queue.cc:1035] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:18.176404 7173 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: TxnStatusTablet state changed. Reason: New leader 391c7b31a76a4affbc49e2e66daea348. Latest consensus state: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.176496 7173 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:18.176501 7177 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: TxnStatusTablet state changed. Reason: New leader 391c7b31a76a4affbc49e2e66daea348. Latest consensus state: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.176605 7177 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:18.177727 7175 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.177827 7175 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:18.177935 7177 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.178027 7177 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P 4393185ad74e4d41aceebf8743319521: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:18.178265 7197 mvcc.cc:205] Tried to move back new op lower bound from 7332666729161482240 to 7332666729125556224. Current Snapshot: MvccSnapshot[applied={T|T < 7332666729161482240}]
I20260923 22:17:18.178762 7191 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.178839 7191 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P 391c7b31a76a4affbc49e2e66daea348: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:18.179054 7173 tablet_replica.cc:439] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } } }
I20260923 22:17:18.179128 7173 tablet_replica.cc:442] T ab7a1ba38f244ddb9040f55fa9f2396e P db89ffe0e31e4e009c3cf74633ab0f45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:18.179756 7198 mvcc.cc:205] Tried to move back new op lower bound from 7332666729161482240 to 7332666729125556224. Current Snapshot: MvccSnapshot[applied={T|T < 7332666729161482240 or (T in {7332666729161482240})}]
I20260923 22:17:18.182919 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.180512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.4.207.1:43731)
0923 22:17:18.180552 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.180554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.180561 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:18.180647 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.180648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.180665 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.180734 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:18.180837 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.181327 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.182105 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.182324 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.182409 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.182574 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.182686 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.182780 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.182798 (+ 18us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.182823 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:18.188387 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.185706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:38135 (local address 127.4.207.3:45601)
0923 22:17:18.185751 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.185753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.185841 (+ 88us) server_negotiation.cc:407] Connection header received
0923 22:17:18.185887 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.185888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.185898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.185947 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:18.186082 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.186598 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.187773 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.187947 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.187999 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.188197 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.188223 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.188260 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.188316 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.188333 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:18.188457 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.185601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45601 (local address 127.4.207.1:38135)
0923 22:17:18.185755 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.185768 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.185830 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.185958 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.185959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.185960 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:18.186023 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.186025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.186673 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.186674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.187680 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.187683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.188016 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.188016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.188083 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.188100 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.188262 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.188263 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.188311 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.188356 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260923 22:17:18.190968 6960 raft_consensus.cc:1261] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Refusing update from remote peer db89ffe0e31e4e009c3cf74633ab0f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:18.191177 7100 raft_consensus.cc:1261] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Refusing update from remote peer db89ffe0e31e4e009c3cf74633ab0f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:18.191247 7179 consensus_queue.cc:1035] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:18.191409 7173 consensus_queue.cc:1035] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:18.193606 7196 mvcc.cc:205] Tried to move back new op lower bound from 7332666729228275712 to 7332666728553803776. Current Snapshot: MvccSnapshot[applied={T|T < 7332666729228275712}]
I20260923 22:17:18.200393 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.197651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:46491 (local address 127.4.207.1:43731)
0923 22:17:18.197686 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.197688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.197698 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:18.197761 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.197762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.197771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.197837 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:18.197978 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.198756 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.199633 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.199935 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.200007 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.200141 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.200176 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.200228 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.200298 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.200317 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:17:18.200425 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.197589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:43731 (local address 127.4.207.3:46491)
0923 22:17:18.197661 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.197672 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.197709 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.197854 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.197855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.197856 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:18.197915 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.197917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.198834 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.198835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.199558 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.199561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.200008 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.200009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.200052 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.200066 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.200240 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.200241 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.200293 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.200337 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 22:17:18.208159 6663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:18.208284 6663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:18.209043 6663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: Sending DeleteTablet for 3 replicas of tablet ff456ad8dc624226b0b213f7d5d27cbe
I20260923 22:17:18.209386 7076 tablet_service.cc:1514] Processing DeleteTablet for tablet ff456ad8dc624226b0b213f7d5d27cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:18 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56028
I20260923 22:17:18.209378 6940 tablet_service.cc:1514] Processing DeleteTablet for tablet ff456ad8dc624226b0b213f7d5d27cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:18 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:41538
I20260923 22:17:18.209627 6800 tablet_service.cc:1514] Processing DeleteTablet for tablet ff456ad8dc624226b0b213f7d5d27cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:18 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56054
I20260923 22:17:18.209971 7216 tablet_replica.cc:330] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: stopping tablet replica
I20260923 22:17:18.210119 7216 raft_consensus.cc:2229] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:18.210047 6663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:18.210201 6663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:18.210382 7216 raft_consensus.cc:2258] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:18.210795 7215 tablet_replica.cc:330] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: stopping tablet replica
I20260923 22:17:18.211239 7216 ts_tablet_manager.cc:1905] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:18.211555 7076 tablet_service.cc:1467] Processing CreateTablet for tablet 05077e99e2e54eedad4b0b004d22347b (DEFAULT_TABLE table=test-table [id=b53a2baed46c4a7a88c96cb78cd00128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:18.211689 7076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05077e99e2e54eedad4b0b004d22347b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.211778 6800 tablet_service.cc:1467] Processing CreateTablet for tablet 05077e99e2e54eedad4b0b004d22347b (DEFAULT_TABLE table=test-table [id=b53a2baed46c4a7a88c96cb78cd00128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:18.211877 6800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05077e99e2e54eedad4b0b004d22347b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.213114 7169 tablet_bootstrap.cc:492] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: Bootstrap starting.
I20260923 22:17:18.213578 7169 tablet_bootstrap.cc:654] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.213902 7170 tablet_bootstrap.cc:492] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: Bootstrap starting.
I20260923 22:17:18.214260 7170 tablet_bootstrap.cc:654] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.214985 7170 tablet_bootstrap.cc:492] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: No bootstrap required, opened a new log
I20260923 22:17:18.215037 7170 ts_tablet_manager.cc:1397] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:18.215184 7170 raft_consensus.cc:348] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.215256 7170 raft_consensus.cc:374] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.215276 7170 raft_consensus.cc:729] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89ffe0e31e4e009c3cf74633ab0f45, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.215343 7170 consensus_queue.cc:260] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.215453 7170 ts_tablet_manager.cc:1428] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.215859 7169 tablet_bootstrap.cc:492] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: No bootstrap required, opened a new log
I20260923 22:17:18.215906 7169 ts_tablet_manager.cc:1397] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 22:17:18.216063 7169 raft_consensus.cc:348] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.216140 7169 raft_consensus.cc:374] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.216161 7169 raft_consensus.cc:729] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391c7b31a76a4affbc49e2e66daea348, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.216202 7169 consensus_queue.cc:260] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.216291 7169 ts_tablet_manager.cc:1428] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:18.211615 6940 tablet_service.cc:1467] Processing CreateTablet for tablet 05077e99e2e54eedad4b0b004d22347b (DEFAULT_TABLE table=test-table [id=b53a2baed46c4a7a88c96cb78cd00128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:18.218087 6940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05077e99e2e54eedad4b0b004d22347b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:18.218504 7216 ts_tablet_manager.cc:1918] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:18.218613 7216 log.cc:1196] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/ff456ad8dc624226b0b213f7d5d27cbe
I20260923 22:17:18.218967 7216 ts_tablet_manager.cc:1939] T ff456ad8dc624226b0b213f7d5d27cbe P 391c7b31a76a4affbc49e2e66daea348: Deleting consensus metadata
I20260923 22:17:18.219612 7174 tablet_bootstrap.cc:492] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: Bootstrap starting.
I20260923 22:17:18.219965 6650 catalog_manager.cc:4841] TS 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601): tablet ff456ad8dc624226b0b213f7d5d27cbe (table test-table [id=c3974c89c21b4806b8d660a2eab5689e]) successfully deleted
I20260923 22:17:18.220029 7174 tablet_bootstrap.cc:654] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:18.220669 7174 tablet_bootstrap.cc:492] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: No bootstrap required, opened a new log
I20260923 22:17:18.220736 7174 ts_tablet_manager.cc:1397] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:18.220885 7174 raft_consensus.cc:348] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.220963 7174 raft_consensus.cc:374] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:18.220990 7174 raft_consensus.cc:729] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4393185ad74e4d41aceebf8743319521, State: Initialized, Role: FOLLOWER
I20260923 22:17:18.221031 7174 consensus_queue.cc:260] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.221124 7174 ts_tablet_manager.cc:1428] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:18.221345 7215 raft_consensus.cc:2229] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:18.221416 7215 raft_consensus.cc:2258] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:18.221728 7217 tablet_replica.cc:330] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: stopping tablet replica
I20260923 22:17:18.221870 7217 raft_consensus.cc:2229] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:18.221978 7217 raft_consensus.cc:2258] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:18.222313 7215 ts_tablet_manager.cc:1905] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:18.223887 7217 ts_tablet_manager.cc:1905] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:18.224035 7215 ts_tablet_manager.cc:1918] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:18.224159 7215 log.cc:1196] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/ff456ad8dc624226b0b213f7d5d27cbe
I20260923 22:17:18.224464 7215 ts_tablet_manager.cc:1939] T ff456ad8dc624226b0b213f7d5d27cbe P 4393185ad74e4d41aceebf8743319521: Deleting consensus metadata
I20260923 22:17:18.224916 6647 catalog_manager.cc:4841] TS 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317): tablet ff456ad8dc624226b0b213f7d5d27cbe (table test-table [id=c3974c89c21b4806b8d660a2eab5689e]) successfully deleted
I20260923 22:17:18.225796 7217 ts_tablet_manager.cc:1918] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:18.225871 7217 log.cc:1196] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/ff456ad8dc624226b0b213f7d5d27cbe
I20260923 22:17:18.226143 7217 ts_tablet_manager.cc:1939] T ff456ad8dc624226b0b213f7d5d27cbe P db89ffe0e31e4e009c3cf74633ab0f45: Deleting consensus metadata
I20260923 22:17:18.226516 6647 catalog_manager.cc:4841] TS db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731): tablet ff456ad8dc624226b0b213f7d5d27cbe (table test-table [id=c3974c89c21b4806b8d660a2eab5689e]) successfully deleted
I20260923 22:17:18.261837 7177 raft_consensus.cc:482] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:18.261953 7177 raft_consensus.cc:504] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.262337 7177 leader_election.cc:288] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731), 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601)
I20260923 22:17:18.266268 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.262485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:43731 (local address 127.4.207.2:36949)
0923 22:17:18.262601 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.262613 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.262670 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.262969 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.262970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.262972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:18.263043 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.263045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.264003 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.264004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.265203 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.265205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.266021 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.266023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.266095 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.266099 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.266148 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.266194 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 22:17:18.266281 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.262561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:59131 (local address 127.4.207.3:45601)
0923 22:17:18.262605 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.262608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.262800 (+ 192us) server_negotiation.cc:407] Connection header received
0923 22:17:18.262912 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.262913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.262925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.262955 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:18.263544 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.264210 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.265435 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.265929 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.265967 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.265978 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.266195 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.266225 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:17:18.266531 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.262486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45601 (local address 127.4.207.2:59131)
0923 22:17:18.262747 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.262757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.262825 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.263013 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.263014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.263016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:18.263424 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.263427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.264302 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.264303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.265351 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.265353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.266060 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.266061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.266118 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.266121 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.266386 (+ 265us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.266430 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":146,"thread_start_us":62,"threads_started":1}
I20260923 22:17:18.266708 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05077e99e2e54eedad4b0b004d22347b" candidate_uuid: "4393185ad74e4d41aceebf8743319521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391c7b31a76a4affbc49e2e66daea348" is_pre_election: true
I20260923 22:17:18.266805 7100 raft_consensus.cc:2454] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4393185ad74e4d41aceebf8743319521 in term 0.
I20260923 22:17:18.266983 6894 leader_election.cc:302] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391c7b31a76a4affbc49e2e66daea348, 4393185ad74e4d41aceebf8743319521; no voters:
I20260923 22:17:18.267019 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.262764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:36949 (local address 127.4.207.1:43731)
0923 22:17:18.262806 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.262808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.262815 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:18.262843 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.262844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.262855 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.262917 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:18.263124 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.263915 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.265375 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.265918 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.265980 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.265990 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.266908 (+ 918us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.266948 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:18.267113 7177 raft_consensus.cc:2781] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:18.267143 7177 raft_consensus.cc:482] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:18.267164 7177 raft_consensus.cc:3037] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.267179 6820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05077e99e2e54eedad4b0b004d22347b" candidate_uuid: "4393185ad74e4d41aceebf8743319521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" is_pre_election: true
I20260923 22:17:18.267267 6820 raft_consensus.cc:2454] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4393185ad74e4d41aceebf8743319521 in term 0.
I20260923 22:17:18.268074 7177 raft_consensus.cc:504] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.268219 7177 leader_election.cc:288] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [CANDIDATE]: Term 1 election: Requested vote from peers db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731), 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601)
I20260923 22:17:18.268369 6820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05077e99e2e54eedad4b0b004d22347b" candidate_uuid: "4393185ad74e4d41aceebf8743319521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89ffe0e31e4e009c3cf74633ab0f45"
I20260923 22:17:18.268392 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05077e99e2e54eedad4b0b004d22347b" candidate_uuid: "4393185ad74e4d41aceebf8743319521" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391c7b31a76a4affbc49e2e66daea348"
I20260923 22:17:18.268433 6820 raft_consensus.cc:3037] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.268450 7100 raft_consensus.cc:3037] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:18.269233 7100 raft_consensus.cc:2454] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4393185ad74e4d41aceebf8743319521 in term 1.
I20260923 22:17:18.269233 6820 raft_consensus.cc:2454] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4393185ad74e4d41aceebf8743319521 in term 1.
I20260923 22:17:18.269399 6894 leader_election.cc:302] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391c7b31a76a4affbc49e2e66daea348, 4393185ad74e4d41aceebf8743319521; no voters:
I20260923 22:17:18.269479 7177 raft_consensus.cc:2781] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:18.269598 7177 raft_consensus.cc:686] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 1 LEADER]: Becoming Leader. State: Replica: 4393185ad74e4d41aceebf8743319521, State: Running, Role: LEADER
I20260923 22:17:18.269672 7177 consensus_queue.cc:237] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:18.270449 6663 catalog_manager.cc:5495] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4393185ad74e4d41aceebf8743319521 (127.4.207.2). New cstate: current_term: 1 leader_uuid: "4393185ad74e4d41aceebf8743319521" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:18.278064 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.275738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.4.207.2:44317)
0923 22:17:18.275773 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.275775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.275804 (+ 29us) server_negotiation.cc:407] Connection header received
0923 22:17:18.275888 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.275889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.275907 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.275992 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:18.276091 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.276547 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.277265 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.277466 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.277537 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.277700 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.277810 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.277911 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.277949 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.277968 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:17:18.282781 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.280523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:37249 (local address 127.4.207.3:45601)
0923 22:17:18.280607 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.280609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.280616 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:18.280637 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.280637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.280654 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.280718 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:18.280840 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.281298 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.282036 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.282271 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.282347 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.282527 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.282561 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.282630 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.282691 (+ 61us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.282709 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 22:17:18.282793 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.280415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45601 (local address 127.4.207.2:37249)
0923 22:17:18.280487 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.280498 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.280538 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.280719 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.280720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.280721 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:18.280775 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.280776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.281379 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.281380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.281950 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.281951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.282345 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.282346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.282411 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.282432 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.282631 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.282634 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.282687 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.282731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 22:17:18.284996 6820 raft_consensus.cc:1261] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Refusing update from remote peer 4393185ad74e4d41aceebf8743319521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:18.285254 7222 consensus_queue.cc:1035] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:18.285683 7100 raft_consensus.cc:1261] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Refusing update from remote peer 4393185ad74e4d41aceebf8743319521: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:18.287034 7227 consensus_queue.cc:1035] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [LEADER]: Connected to new peer: Peer: permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:18.293758 7168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.290855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:43739 (local address 127.4.207.2:44317)
0923 22:17:18.291021 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:18.291023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:18.291030 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:18.291064 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:18.291064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:18.291075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:18.291124 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:18.291292 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.292003 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.293021 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.293305 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.293367 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.293536 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.293571 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.293614 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:18.293672 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 22:17:18.293692 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260923 22:17:18.293787 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:18.290665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:44317 (local address 127.4.207.3:43739)
0923 22:17:18.290742 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:18.290754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:18.290790 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:18.291172 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:18.291173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:18.291174 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:18.291218 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.291220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.292134 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.292135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.292925 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:18.292928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:18.293398 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:18.293399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:18.293457 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:18.293462 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:18.293627 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:18.293629 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:18.293672 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 22:17:18.293715 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
W20260923 22:17:18.294325 7197 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 22:17:19.818120 6663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:19.818274 6663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:19.819115 6663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: Sending DeleteTablet for 3 replicas of tablet 05077e99e2e54eedad4b0b004d22347b
I20260923 22:17:19.819329 6800 tablet_service.cc:1514] Processing DeleteTablet for tablet 05077e99e2e54eedad4b0b004d22347b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:19 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56054
I20260923 22:17:19.819375 6940 tablet_service.cc:1514] Processing DeleteTablet for tablet 05077e99e2e54eedad4b0b004d22347b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:19 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:41538
I20260923 22:17:19.821180 7076 tablet_service.cc:1514] Processing DeleteTablet for tablet 05077e99e2e54eedad4b0b004d22347b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:19 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56028
I20260923 22:17:19.823546 7254 tablet_replica.cc:330] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: stopping tablet replica
I20260923 22:17:19.823644 7254 raft_consensus.cc:2229] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:19.823693 7254 raft_consensus.cc:2258] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:19.823760 7251 tablet_replica.cc:330] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: stopping tablet replica
I20260923 22:17:19.823829 7252 tablet_replica.cc:330] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: stopping tablet replica
I20260923 22:17:19.823858 7251 raft_consensus.cc:2229] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:19.823900 7251 raft_consensus.cc:2258] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:19.824060 7254 ts_tablet_manager.cc:1905] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:19.824170 7252 raft_consensus.cc:2229] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:19.824198 7251 ts_tablet_manager.cc:1905] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:19.824286 7252 raft_consensus.cc:2258] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:19.826136 7252 ts_tablet_manager.cc:1905] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:19.826179 7251 ts_tablet_manager.cc:1918] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:19.826179 7254 ts_tablet_manager.cc:1918] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:19.826227 7251 log.cc:1196] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/05077e99e2e54eedad4b0b004d22347b
I20260923 22:17:19.826444 7251 ts_tablet_manager.cc:1939] T 05077e99e2e54eedad4b0b004d22347b P db89ffe0e31e4e009c3cf74633ab0f45: Deleting consensus metadata
I20260923 22:17:19.826848 6647 catalog_manager.cc:4841] TS db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731): tablet 05077e99e2e54eedad4b0b004d22347b (table test-table [id=b53a2baed46c4a7a88c96cb78cd00128]) successfully deleted
I20260923 22:17:19.827828 7252 ts_tablet_manager.cc:1918] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:19.827877 7252 log.cc:1196] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/05077e99e2e54eedad4b0b004d22347b
I20260923 22:17:19.828114 7252 ts_tablet_manager.cc:1939] T 05077e99e2e54eedad4b0b004d22347b P 4393185ad74e4d41aceebf8743319521: Deleting consensus metadata
I20260923 22:17:19.828450 6647 catalog_manager.cc:4841] TS 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317): tablet 05077e99e2e54eedad4b0b004d22347b (table test-table [id=b53a2baed46c4a7a88c96cb78cd00128]) successfully deleted
I20260923 22:17:19.830205 7254 log.cc:1196] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/05077e99e2e54eedad4b0b004d22347b
I20260923 22:17:19.830456 7254 ts_tablet_manager.cc:1939] T 05077e99e2e54eedad4b0b004d22347b P 391c7b31a76a4affbc49e2e66daea348: Deleting consensus metadata
Sep 23 22:17:19 dist-test-slave-7511 krb5kdc[6618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260923 22:17:19.830828 6650 catalog_manager.cc:4841] TS 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601): tablet 05077e99e2e54eedad4b0b004d22347b (table test-table [id=b53a2baed46c4a7a88c96cb78cd00128]) successfully deleted
Sep 23 22:17:19 dist-test-slave-7511 krb5kdc[6618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201839, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:19.843858 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:19.836394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.4.207.62:35323)
0923 22:17:19.836546 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:19.836548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:19.836562 (+ 14us) server_negotiation.cc:407] Connection header received
0923 22:17:19.836679 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:19.836679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:19.836690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:19.836753 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:19.837520 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:19.838053 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:19.838789 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:19.838983 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:19.839033 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:19.841375 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:19.841376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:19.841377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:19.841384 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:19.842845 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:19.843134 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:19.843134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:19.843135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:19.843184 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:19.843326 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:19.843327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:19.843327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:19.843423 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:19.843590 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:19.843721 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 22:17:19.843786 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":103,"threads_started":1}
I20260923 22:17:19.845427 6663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:19.845608 6663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:19.847012 6800 tablet_service.cc:1467] Processing CreateTablet for tablet 46cf35817913404db16a83b85bfb3493 (DEFAULT_TABLE table=test-table [id=6402345c516f4a6cb15845a9224f4a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:19.847060 7076 tablet_service.cc:1467] Processing CreateTablet for tablet 46cf35817913404db16a83b85bfb3493 (DEFAULT_TABLE table=test-table [id=6402345c516f4a6cb15845a9224f4a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:19.847043 6940 tablet_service.cc:1467] Processing CreateTablet for tablet 46cf35817913404db16a83b85bfb3493 (DEFAULT_TABLE table=test-table [id=6402345c516f4a6cb15845a9224f4a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:19.847167 7076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cf35817913404db16a83b85bfb3493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:19.847170 6800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cf35817913404db16a83b85bfb3493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:19.847223 6940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cf35817913404db16a83b85bfb3493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:19.848346 7262 tablet_bootstrap.cc:492] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: Bootstrap starting.
I20260923 22:17:19.848670 7264 tablet_bootstrap.cc:492] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: Bootstrap starting.
I20260923 22:17:19.848773 7262 tablet_bootstrap.cc:654] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:19.849026 7264 tablet_bootstrap.cc:654] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:19.849362 7263 tablet_bootstrap.cc:492] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: Bootstrap starting.
I20260923 22:17:19.849447 7262 tablet_bootstrap.cc:492] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: No bootstrap required, opened a new log
I20260923 22:17:19.849503 7262 ts_tablet_manager.cc:1397] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:19.849697 7264 tablet_bootstrap.cc:492] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: No bootstrap required, opened a new log
I20260923 22:17:19.849656 7262 raft_consensus.cc:348] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.849737 7262 raft_consensus.cc:374] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:19.849752 7264 ts_tablet_manager.cc:1397] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:19.849761 7262 raft_consensus.cc:729] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4393185ad74e4d41aceebf8743319521, State: Initialized, Role: FOLLOWER
I20260923 22:17:19.849727 7263 tablet_bootstrap.cc:654] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:19.849833 7262 consensus_queue.cc:260] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.849864 7264 raft_consensus.cc:348] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.849947 7264 raft_consensus.cc:374] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:19.850025 7264 raft_consensus.cc:729] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89ffe0e31e4e009c3cf74633ab0f45, State: Initialized, Role: FOLLOWER
I20260923 22:17:19.850095 7262 ts_tablet_manager.cc:1428] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:19.850117 7264 consensus_queue.cc:260] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.850322 7264 ts_tablet_manager.cc:1428] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:19.850792 7263 tablet_bootstrap.cc:492] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: No bootstrap required, opened a new log
I20260923 22:17:19.850847 7263 ts_tablet_manager.cc:1397] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:19.850955 7263 raft_consensus.cc:348] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.851022 7263 raft_consensus.cc:374] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:19.851043 7263 raft_consensus.cc:729] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391c7b31a76a4affbc49e2e66daea348, State: Initialized, Role: FOLLOWER
I20260923 22:17:19.851101 7263 consensus_queue.cc:260] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.851270 7263 ts_tablet_manager.cc:1428] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260923 22:17:19.957813 7147 tablet.cc:2369] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:19.976667 7232 raft_consensus.cc:482] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:19.976773 7232 raft_consensus.cc:504] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.976936 7232 leader_election.cc:288] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731), 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317)
I20260923 22:17:19.977240 6820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cf35817913404db16a83b85bfb3493" candidate_uuid: "391c7b31a76a4affbc49e2e66daea348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" is_pre_election: true
I20260923 22:17:19.977272 6960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cf35817913404db16a83b85bfb3493" candidate_uuid: "391c7b31a76a4affbc49e2e66daea348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393185ad74e4d41aceebf8743319521" is_pre_election: true
I20260923 22:17:19.977351 6960 raft_consensus.cc:2454] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391c7b31a76a4affbc49e2e66daea348 in term 0.
I20260923 22:17:19.977367 6820 raft_consensus.cc:2454] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 391c7b31a76a4affbc49e2e66daea348 in term 0.
I20260923 22:17:19.977578 7031 leader_election.cc:302] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391c7b31a76a4affbc49e2e66daea348, 4393185ad74e4d41aceebf8743319521; no voters:
I20260923 22:17:19.977717 7232 raft_consensus.cc:2781] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:19.977769 7232 raft_consensus.cc:482] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:19.977793 7232 raft_consensus.cc:3037] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:19.978606 7232 raft_consensus.cc:504] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.978713 7232 leader_election.cc:288] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [CANDIDATE]: Term 1 election: Requested vote from peers db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731), 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317)
I20260923 22:17:19.978958 6820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cf35817913404db16a83b85bfb3493" candidate_uuid: "391c7b31a76a4affbc49e2e66daea348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db89ffe0e31e4e009c3cf74633ab0f45"
I20260923 22:17:19.978995 6960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cf35817913404db16a83b85bfb3493" candidate_uuid: "391c7b31a76a4affbc49e2e66daea348" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393185ad74e4d41aceebf8743319521"
I20260923 22:17:19.979065 6960 raft_consensus.cc:3037] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:19.979087 6820 raft_consensus.cc:3037] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:19.979857 6960 raft_consensus.cc:2454] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391c7b31a76a4affbc49e2e66daea348 in term 1.
I20260923 22:17:19.979920 6820 raft_consensus.cc:2454] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 391c7b31a76a4affbc49e2e66daea348 in term 1.
I20260923 22:17:19.980038 7031 leader_election.cc:302] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 391c7b31a76a4affbc49e2e66daea348, 4393185ad74e4d41aceebf8743319521; no voters:
I20260923 22:17:19.980154 7232 raft_consensus.cc:2781] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:19.980247 7232 raft_consensus.cc:686] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 1 LEADER]: Becoming Leader. State: Replica: 391c7b31a76a4affbc49e2e66daea348, State: Running, Role: LEADER
I20260923 22:17:19.980302 7232 consensus_queue.cc:237] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:19.981027 6661 catalog_manager.cc:5495] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 reported cstate change: term changed from 0 to 1, leader changed from <none> to 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3). New cstate: current_term: 1 leader_uuid: "391c7b31a76a4affbc49e2e66daea348" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } health_report { overall_health: HEALTHY } } }
I20260923 22:17:19.999764 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:19.997250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.4.207.3:45601)
0923 22:17:19.997425 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:19.997427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:19.997434 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:19.997466 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:19.997467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:19.997476 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:19.997552 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:19.997715 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:19.998216 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:19.998985 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:19.999215 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:19.999295 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:19.999482 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:19.999517 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:19.999586 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:19.999645 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 22:17:19.999669 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260923 22:17:20.004577 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:20.002200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:51001 (local address 127.4.207.3:45601)
0923 22:17:20.002253 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:20.002255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:20.002281 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:17:20.002355 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:20.002355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:20.002364 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:20.002413 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:20.002515 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:20.003084 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:20.003883 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:20.004082 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:20.004137 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:20.004326 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:20.004371 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:20.004426 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:20.004475 (+ 49us) server_negotiation.cc:299] Negotiation successful
0923 22:17:20.004500 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:17:20.004585 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:20.002040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:45601 (local address 127.4.207.3:51001)
0923 22:17:20.002222 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:20.002235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:20.002303 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:20.002427 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:20.002428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:20.002429 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:20.002466 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:20.002467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:20.003183 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:20.003184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:20.003816 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:20.003818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:20.004147 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:20.004148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:20.004205 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:20.004209 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:20.004434 (+ 225us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:20.004435 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:20.004466 (+ 31us) client_negotiation.cc:240] Negotiation successful
0923 22:17:20.004501 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260923 22:17:20.007079 6820 raft_consensus.cc:1261] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Refusing update from remote peer 391c7b31a76a4affbc49e2e66daea348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:20.007087 6960 raft_consensus.cc:1261] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Refusing update from remote peer 391c7b31a76a4affbc49e2e66daea348: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:20.007323 7210 consensus_queue.cc:1035] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:20.007521 7237 consensus_queue.cc:1035] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:20.017656 6662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:20.017776 6662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:20.018491 6662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: Sending DeleteTablet for 3 replicas of tablet 46cf35817913404db16a83b85bfb3493
I20260923 22:17:20.018733 6800 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cf35817913404db16a83b85bfb3493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:20 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56054
I20260923 22:17:20.018791 7076 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cf35817913404db16a83b85bfb3493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:20 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56028
I20260923 22:17:20.018784 6940 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cf35817913404db16a83b85bfb3493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:20 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:41538
I20260923 22:17:20.018821 7251 tablet_replica.cc:330] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: stopping tablet replica
I20260923 22:17:20.018872 7251 raft_consensus.cc:2229] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:20.018880 7252 tablet_replica.cc:330] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: stopping tablet replica
I20260923 22:17:20.018879 7254 tablet_replica.cc:330] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: stopping tablet replica
I20260923 22:17:20.018908 7251 raft_consensus.cc:2258] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:20.018922 7252 raft_consensus.cc:2229] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:20.018929 7254 raft_consensus.cc:2229] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:20.018959 7252 raft_consensus.cc:2258] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:20.019013 7254 raft_consensus.cc:2258] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:20.019364 7252 ts_tablet_manager.cc:1905] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:20.019364 7254 ts_tablet_manager.cc:1905] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:20.019374 7251 ts_tablet_manager.cc:1905] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:20.021063 7254 ts_tablet_manager.cc:1918] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:20.021138 7254 log.cc:1196] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/46cf35817913404db16a83b85bfb3493
I20260923 22:17:20.021240 7251 ts_tablet_manager.cc:1918] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:20.021291 7251 log.cc:1196] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/46cf35817913404db16a83b85bfb3493
I20260923 22:17:20.021207 6662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:20.021358 6662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:20.021364 7254 ts_tablet_manager.cc:1939] T 46cf35817913404db16a83b85bfb3493 P 391c7b31a76a4affbc49e2e66daea348: Deleting consensus metadata
I20260923 22:17:20.021502 7251 ts_tablet_manager.cc:1939] T 46cf35817913404db16a83b85bfb3493 P db89ffe0e31e4e009c3cf74633ab0f45: Deleting consensus metadata
I20260923 22:17:20.021799 7252 ts_tablet_manager.cc:1918] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:20.021871 7252 log.cc:1196] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/46cf35817913404db16a83b85bfb3493
I20260923 22:17:20.021966 6647 catalog_manager.cc:4841] TS db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731): tablet 46cf35817913404db16a83b85bfb3493 (table test-table [id=6402345c516f4a6cb15845a9224f4a3d]) successfully deleted
I20260923 22:17:20.022081 7252 ts_tablet_manager.cc:1939] T 46cf35817913404db16a83b85bfb3493 P 4393185ad74e4d41aceebf8743319521: Deleting consensus metadata
I20260923 22:17:20.022547 6647 catalog_manager.cc:4841] TS 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317): tablet 46cf35817913404db16a83b85bfb3493 (table test-table [id=6402345c516f4a6cb15845a9224f4a3d]) successfully deleted
I20260923 22:17:20.022745 6650 catalog_manager.cc:4841] TS 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601): tablet 46cf35817913404db16a83b85bfb3493 (table test-table [id=6402345c516f4a6cb15845a9224f4a3d]) successfully deleted
I20260923 22:17:20.022846 7076 tablet_service.cc:1467] Processing CreateTablet for tablet c896277807ec450db44e3beb21887a78 (DEFAULT_TABLE table=test-table [id=e029ab18d413442bafdfea306ff011b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:20.022847 6800 tablet_service.cc:1467] Processing CreateTablet for tablet c896277807ec450db44e3beb21887a78 (DEFAULT_TABLE table=test-table [id=e029ab18d413442bafdfea306ff011b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:20.022948 7076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896277807ec450db44e3beb21887a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:20.022948 6800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896277807ec450db44e3beb21887a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:20.023874 7263 tablet_bootstrap.cc:492] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: Bootstrap starting.
I20260923 22:17:20.024143 6940 tablet_service.cc:1467] Processing CreateTablet for tablet c896277807ec450db44e3beb21887a78 (DEFAULT_TABLE table=test-table [id=e029ab18d413442bafdfea306ff011b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:20.024222 7263 tablet_bootstrap.cc:654] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:20.024240 6940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c896277807ec450db44e3beb21887a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:20.024447 7264 tablet_bootstrap.cc:492] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45: Bootstrap starting.
I20260923 22:17:20.024730 7264 tablet_bootstrap.cc:654] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:20.025250 7264 tablet_bootstrap.cc:492] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45: No bootstrap required, opened a new log
I20260923 22:17:20.025293 7264 ts_tablet_manager.cc:1397] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:20.025393 7264 raft_consensus.cc:348] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.025427 7262 tablet_bootstrap.cc:492] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: Bootstrap starting.
I20260923 22:17:20.025457 7264 raft_consensus.cc:374] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:20.025478 7264 raft_consensus.cc:729] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db89ffe0e31e4e009c3cf74633ab0f45, State: Initialized, Role: FOLLOWER
I20260923 22:17:20.025528 7264 consensus_queue.cc:260] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.025628 7264 ts_tablet_manager.cc:1428] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:20.025735 7262 tablet_bootstrap.cc:654] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:20.026000 7263 tablet_bootstrap.cc:492] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: No bootstrap required, opened a new log
I20260923 22:17:20.026052 7263 ts_tablet_manager.cc:1397] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:20.026149 7263 raft_consensus.cc:348] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.026234 7263 raft_consensus.cc:374] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:20.026265 7263 raft_consensus.cc:729] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391c7b31a76a4affbc49e2e66daea348, State: Initialized, Role: FOLLOWER
I20260923 22:17:20.026328 7263 consensus_queue.cc:260] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.026477 7263 ts_tablet_manager.cc:1428] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:20.026598 7262 tablet_bootstrap.cc:492] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: No bootstrap required, opened a new log
I20260923 22:17:20.026640 7262 ts_tablet_manager.cc:1397] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:20.026742 7262 raft_consensus.cc:348] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.026799 7262 raft_consensus.cc:374] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:20.026818 7262 raft_consensus.cc:729] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4393185ad74e4d41aceebf8743319521, State: Initialized, Role: FOLLOWER
I20260923 22:17:20.026865 7262 consensus_queue.cc:260] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.026944 7262 ts_tablet_manager.cc:1428] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:20.038262 7268 raft_consensus.cc:482] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:20.038378 7268 raft_consensus.cc:504] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.038578 7268 leader_election.cc:288] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317), 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601)
I20260923 22:17:20.038777 6960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896277807ec450db44e3beb21887a78" candidate_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393185ad74e4d41aceebf8743319521" is_pre_election: true
I20260923 22:17:20.038769 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896277807ec450db44e3beb21887a78" candidate_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391c7b31a76a4affbc49e2e66daea348" is_pre_election: true
I20260923 22:17:20.038872 6960 raft_consensus.cc:2454] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89ffe0e31e4e009c3cf74633ab0f45 in term 0.
I20260923 22:17:20.038892 7100 raft_consensus.cc:2454] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db89ffe0e31e4e009c3cf74633ab0f45 in term 0.
I20260923 22:17:20.039033 6751 leader_election.cc:302] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4393185ad74e4d41aceebf8743319521, db89ffe0e31e4e009c3cf74633ab0f45; no voters:
I20260923 22:17:20.039144 7268 raft_consensus.cc:2781] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:20.039179 7268 raft_consensus.cc:482] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:20.039199 7268 raft_consensus.cc:3037] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:20.040150 7268 raft_consensus.cc:504] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.040300 7268 leader_election.cc:288] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [CANDIDATE]: Term 1 election: Requested vote from peers 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317), 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601)
I20260923 22:17:20.040552 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896277807ec450db44e3beb21887a78" candidate_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "391c7b31a76a4affbc49e2e66daea348"
I20260923 22:17:20.040545 6960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c896277807ec450db44e3beb21887a78" candidate_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4393185ad74e4d41aceebf8743319521"
I20260923 22:17:20.040637 7100 raft_consensus.cc:3037] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:20.040654 6960 raft_consensus.cc:3037] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:20.041266 6960 raft_consensus.cc:2454] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89ffe0e31e4e009c3cf74633ab0f45 in term 1.
I20260923 22:17:20.041364 7100 raft_consensus.cc:2454] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db89ffe0e31e4e009c3cf74633ab0f45 in term 1.
I20260923 22:17:20.041461 6751 leader_election.cc:302] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4393185ad74e4d41aceebf8743319521, db89ffe0e31e4e009c3cf74633ab0f45; no voters:
I20260923 22:17:20.041576 7268 raft_consensus.cc:2781] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:20.041697 7268 raft_consensus.cc:686] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [term 1 LEADER]: Becoming Leader. State: Replica: db89ffe0e31e4e009c3cf74633ab0f45, State: Running, Role: LEADER
I20260923 22:17:20.041752 7268 consensus_queue.cc:237] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } }
I20260923 22:17:20.042400 6661 catalog_manager.cc:5495] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 reported cstate change: term changed from 0 to 1, leader changed from <none> to db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1). New cstate: current_term: 1 leader_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db89ffe0e31e4e009c3cf74633ab0f45" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 43731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:20.053627 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:20.051024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.4.207.1:43731)
0923 22:17:20.051216 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:20.051218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:20.051227 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:20.051255 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:20.051256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:20.051266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:20.051325 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:20.051486 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:20.051977 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:20.053009 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:20.053232 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:20.053297 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:20.053433 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:20.053460 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:20.053501 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:20.053546 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 22:17:20.053562 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":145,"threads_started":1}
I20260923 22:17:20.056128 7100 raft_consensus.cc:1261] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Refusing update from remote peer db89ffe0e31e4e009c3cf74633ab0f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:20.056228 6960 raft_consensus.cc:1261] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Refusing update from remote peer db89ffe0e31e4e009c3cf74633ab0f45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:20.056533 7268 consensus_queue.cc:1035] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "391c7b31a76a4affbc49e2e66daea348" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:20.056671 7268 consensus_queue.cc:1035] T c896277807ec450db44e3beb21887a78 P db89ffe0e31e4e009c3cf74633ab0f45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4393185ad74e4d41aceebf8743319521" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:17:20.066609 7234 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 22:17:21.588354 6661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:21.588478 6661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:21.589330 6661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2d34d28518492888f315de31b22b60: Sending DeleteTablet for 3 replicas of tablet c896277807ec450db44e3beb21887a78
I20260923 22:17:21.589620 6800 tablet_service.cc:1514] Processing DeleteTablet for tablet c896277807ec450db44e3beb21887a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:21 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56054
I20260923 22:17:21.589630 7076 tablet_service.cc:1514] Processing DeleteTablet for tablet c896277807ec450db44e3beb21887a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:21 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56028
I20260923 22:17:21.589644 6940 tablet_service.cc:1514] Processing DeleteTablet for tablet c896277807ec450db44e3beb21887a78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:21 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:41538
I20260923 22:17:21.589797 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 6735
I20260923 22:17:21.590655 7311 tablet_replica.cc:330] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: stopping tablet replica
I20260923 22:17:21.590725 7311 raft_consensus.cc:2229] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:21.590777 7311 raft_consensus.cc:2258] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:21.590929 7310 tablet_replica.cc:330] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: stopping tablet replica
I20260923 22:17:21.590996 7310 raft_consensus.cc:2229] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:21.591034 7310 raft_consensus.cc:2258] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:21.591114 7311 ts_tablet_manager.cc:1905] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:21.591341 7310 ts_tablet_manager.cc:1905] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:21.593045 7311 ts_tablet_manager.cc:1918] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:21.593098 7311 log.cc:1196] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/c896277807ec450db44e3beb21887a78
I20260923 22:17:21.593346 7311 ts_tablet_manager.cc:1939] T c896277807ec450db44e3beb21887a78 P 4393185ad74e4d41aceebf8743319521: Deleting consensus metadata
I20260923 22:17:21.593333 7310 ts_tablet_manager.cc:1918] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 22:17:21.593393 7310 log.cc:1196] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/c896277807ec450db44e3beb21887a78
I20260923 22:17:21.593612 7310 ts_tablet_manager.cc:1939] T c896277807ec450db44e3beb21887a78 P 391c7b31a76a4affbc49e2e66daea348: Deleting consensus metadata
I20260923 22:17:21.593778 6647 catalog_manager.cc:4841] TS 4393185ad74e4d41aceebf8743319521 (127.4.207.2:44317): tablet c896277807ec450db44e3beb21887a78 (table test-table [id=e029ab18d413442bafdfea306ff011b1]) successfully deleted
I20260923 22:17:21.594008 6650 catalog_manager.cc:4841] TS 391c7b31a76a4affbc49e2e66daea348 (127.4.207.3:45601): tablet c896277807ec450db44e3beb21887a78 (table test-table [id=e029ab18d413442bafdfea306ff011b1]) successfully deleted
W20260923 22:17:21.597113 6647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:43731 (error 108)
I20260923 22:17:21.597719 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 6875
I20260923 22:17:21.597739 7314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:21.597538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:43731 (local address 127.0.0.1:52348)
0923 22:17:21.597664 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:21.597704 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:43731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
W20260923 22:17:21.597849 6647 catalog_manager.cc:4584] TS db89ffe0e31e4e009c3cf74633ab0f45 (127.4.207.1:43731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c896277807ec450db44e3beb21887a78: Network error: Client connection negotiation failed: client connection to 127.4.207.1:43731: connect: Connection refused (error 111)
I20260923 22:17:21.603714 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 7015
I20260923 22:17:21.609719 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 6631
2026-09-23T22:17:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5432 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:21 dist-test-slave-7511 krb5kdc[7317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:21 dist-test-slave-7511 krb5kdc[7317](info): set up 2 sockets
Sep 23 22:17:21 dist-test-slave-7511 krb5kdc[7317](info): commencing operation
krb5kdc: starting...
W20260923 22:17:22.659271 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:22 dist-test-slave-7511 krb5kdc[7317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:22.778574 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:43497
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:43497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:22.859488 7330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:22.859746 7330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:22.859787 7330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:22.860975 7330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:22.861050 7330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:22.861076 7330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:22.861104 7330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:22.861147 7330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:22.862636 7330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:43497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:43497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:22.862931 7330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:22.863173 7330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:22.868182 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:22.868170 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:22.868170 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:22.868450 7330 server_base.cc:1034] running on GCE node
I20260923 22:17:22.868722 7330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:22.869014 7330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:22.870239 7330 hybrid_clock.cc:648] HybridClock initialized: now 1790201842870226 us; error 52 us; skew 500 ppm
Sep 23 22:17:22 dist-test-slave-7511 krb5kdc[7317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201842, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:22.872918 7330 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:22.873427 7330 webserver.cc:460] Webserver started at http://127.4.207.62:44405/ using document root <none> and password file <none>
I20260923 22:17:22.873659 7330 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:22.873731 7330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:22.873888 7330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:22.874964 7330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "4855e740612b47d1a386a8407f3f5381"
format_stamp: "Formatted at 2026-09-23 22:17:22 on dist-test-slave-7511"
server_key: "7f1a029892eafc4ef99c0dbf609daaa1"
server_key_iv: "f30ddce10034b05a42c927a0a18dcbfd"
server_key_version: "encryptionkey@0"
I20260923 22:17:22.875344 7330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "4855e740612b47d1a386a8407f3f5381"
format_stamp: "Formatted at 2026-09-23 22:17:22 on dist-test-slave-7511"
server_key: "7f1a029892eafc4ef99c0dbf609daaa1"
server_key_iv: "f30ddce10034b05a42c927a0a18dcbfd"
server_key_version: "encryptionkey@0"
I20260923 22:17:22.876693 7330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:22.877625 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:22.877835 7330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:22.877933 7330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "4855e740612b47d1a386a8407f3f5381"
format_stamp: "Formatted at 2026-09-23 22:17:22 on dist-test-slave-7511"
server_key: "7f1a029892eafc4ef99c0dbf609daaa1"
server_key_iv: "f30ddce10034b05a42c927a0a18dcbfd"
server_key_version: "encryptionkey@0"
I20260923 22:17:22.878031 7330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:22.895664 7330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:22.898226 7330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:22.898411 7330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:22.902640 7330 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:43497
I20260923 22:17:22.902683 7397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:43497 every 8 connection(s)
I20260923 22:17:22.903047 7330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:22.903644 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 7330
I20260923 22:17:22.903748 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:22.903921 4924 external_mini_cluster.cc:1428] Setting key 553028b2b8c0d664d3b627954ab7808b
I20260923 22:17:22.904259 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:22.907047 7398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381: Bootstrap starting.
I20260923 22:17:22.907829 7398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:22.908128 7398 log.cc:824] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:22.908838 7398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381: No bootstrap required, opened a new log
Sep 23 22:17:22 dist-test-slave-7511 krb5kdc[7317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:22.911475 7398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4855e740612b47d1a386a8407f3f5381" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43497 } }
I20260923 22:17:22.911648 7398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:22.911696 7398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4855e740612b47d1a386a8407f3f5381, State: Initialized, Role: FOLLOWER
I20260923 22:17:22.911870 7398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4855e740612b47d1a386a8407f3f5381" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43497 } }
I20260923 22:17:22.911964 7398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:22.912009 7398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:22.912076 7398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:22.912622 7398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4855e740612b47d1a386a8407f3f5381" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43497 } }
I20260923 22:17:22.912742 7398 leader_election.cc:302] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4855e740612b47d1a386a8407f3f5381; no voters:
I20260923 22:17:22.912948 7398 leader_election.cc:288] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:22.912981 7403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:22.913098 7403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [term 1 LEADER]: Becoming Leader. State: Replica: 4855e740612b47d1a386a8407f3f5381, State: Running, Role: LEADER
I20260923 22:17:22.913245 7398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:22.913239 7403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4855e740612b47d1a386a8407f3f5381" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43497 } }
I20260923 22:17:22.914202 7404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4855e740612b47d1a386a8407f3f5381. Latest consensus state: current_term: 1 leader_uuid: "4855e740612b47d1a386a8407f3f5381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4855e740612b47d1a386a8407f3f5381" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43497 } } }
I20260923 22:17:22.914292 7404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:22.914553 7405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4855e740612b47d1a386a8407f3f5381" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4855e740612b47d1a386a8407f3f5381" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43497 } } }
I20260923 22:17:22.914760 7405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381 [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:22.915658 7418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:22.915726 7418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:22.915736 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:22.904430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.4.207.62:43497)
0923 22:17:22.904640 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:22.904649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:22.904685 (+ 36us) server_negotiation.cc:407] Connection header received
0923 22:17:22.905327 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:22.905334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:22.905384 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:22.905532 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:22.906168 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:22.907075 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:22.908309 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:22.908839 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:22.908965 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:22.911578 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:22.911582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:22.911582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:22.911593 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:22.913673 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:22.914018 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:22.914021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:22.914022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:22.914145 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:22.914414 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:22.914414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:22.914415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:22.914750 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:22.915298 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:22.915371 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 22:17:22.915537 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260923 22:17:22.915827 7414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:22.915997 7414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:22.917529 7414 catalog_manager.cc:1347] Generated new cluster ID: 39f6fe263bd1420f91c34546939e5e95
I20260923 22:17:22.917584 7414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:22.922758 7414 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:22.923280 7414 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:22.928593 7414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4855e740612b47d1a386a8407f3f5381: Generated new TSK 0
I20260923 22:17:22.928797 7414 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:22.960537 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43497
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:23.051750 7424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:23.051964 7424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:23.051991 7424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:23.053205 7424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:23.053249 7424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:23.053278 7424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:23.054744 7424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:23.055030 7424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:23.055279 7424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:23.061313 7429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:23.061352 7430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:23.061334 7432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:23.061728 7424 server_base.cc:1034] running on GCE node
I20260923 22:17:23.061946 7424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:23.062227 7424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:23.063452 7424 hybrid_clock.cc:648] HybridClock initialized: now 1790201843063422 us; error 37 us; skew 500 ppm
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:23.066272 7424 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:23.066706 7424 webserver.cc:460] Webserver started at http://127.4.207.1:33661/ using document root <none> and password file <none>
I20260923 22:17:23.066918 7424 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:23.066960 7424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:23.067054 7424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:23.068233 7424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "a2e991e75c0040e58fd501e893e2a4c4"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "fa2f4136fdf55845349b6e7309d3c514"
server_key_iv: "d724e58c48cc38364668810426da817e"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.068544 7424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "a2e991e75c0040e58fd501e893e2a4c4"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "fa2f4136fdf55845349b6e7309d3c514"
server_key_iv: "d724e58c48cc38364668810426da817e"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.069861 7424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:23.070690 7439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.070900 7424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:23.070991 7424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "a2e991e75c0040e58fd501e893e2a4c4"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "fa2f4136fdf55845349b6e7309d3c514"
server_key_iv: "d724e58c48cc38364668810426da817e"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.071046 7424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:23.097703 7424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:23.100155 7424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:23.100282 7424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:23.100488 7424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:23.100832 7424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:23.100869 7424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.100914 7424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:23.100943 7424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.106763 7424 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:36897
I20260923 22:17:23.106823 7552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:36897 every 8 connection(s)
I20260923 22:17:23.107177 7424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:23.116722 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 7424
I20260923 22:17:23.116822 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:23.116823 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:23.107988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:52063 (local address 127.4.207.62:43497)
0923 22:17:23.108035 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:23.108037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:23.108290 (+ 253us) server_negotiation.cc:407] Connection header received
0923 22:17:23.108756 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:23.108756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:23.108766 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:23.108820 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:23.109748 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.110249 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.111225 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.111423 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.111554 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:23.113972 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:23.113974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:23.113974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:23.113981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:23.115411 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:23.115850 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:23.115851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:23.115851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:23.115905 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:23.116119 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:23.116119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:23.116120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:23.116229 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:23.116279 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:23.116578 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 22:17:23.116705 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:23.116843 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:23.107963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43497 (local address 127.4.207.1:52063)
0923 22:17:23.108145 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:23.108200 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:23.108671 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:23.108877 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:23.108880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:23.109458 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:23.109661 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:23.109666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.110333 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.110334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:23.111136 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:23.111138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.111567 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.111568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:23.111671 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:23.112272 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:23.112280 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:23.113880 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:23.115545 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:23.115552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:23.115554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:23.115778 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:23.115966 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:23.115967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:23.115967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:23.116067 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:23.116283 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:23.116287 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:23.116412 (+ 125us) client_negotiation.cc:769] Sending connection context
0923 22:17:23.116497 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 22:17:23.116629 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
I20260923 22:17:23.117013 4924 external_mini_cluster.cc:1428] Setting key d0056b1cd7df726f1eb1445923f9ef3e
I20260923 22:17:23.117759 7553 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43497
I20260923 22:17:23.117898 7553 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:23.118093 7553 heartbeater.cc:511] Master 127.4.207.62:43497 requested a full tablet report, sending...
I20260923 22:17:23.118657 7362 ts_manager.cc:194] Registered new tserver with Master: a2e991e75c0040e58fd501e893e2a4c4 (127.4.207.1:36897)
I20260923 22:17:23.119406 7362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:52063
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:23.143069 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43497
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:23.224743 7558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:23.224961 7558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:23.224987 7558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:23.226194 7558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:23.226238 7558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:23.226266 7558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:23.227782 7558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:23.228040 7558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:23.228288 7558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:23.233464 7564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:23.233493 7563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:23.233481 7566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:23.233788 7558 server_base.cc:1034] running on GCE node
I20260923 22:17:23.233971 7558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:23.234184 7558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:23.235322 7558 hybrid_clock.cc:648] HybridClock initialized: now 1790201843235311 us; error 29 us; skew 500 ppm
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:23.237790 7558 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:23.238224 7558 webserver.cc:460] Webserver started at http://127.4.207.2:41551/ using document root <none> and password file <none>
I20260923 22:17:23.238464 7558 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:23.238530 7558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:23.238669 7558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:23.239828 7558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "5cc8f004a331463a887b2bff89ed6dc6"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "902e56a2d1a923707d88e262c8037f0c"
server_key_iv: "edfd3c65e1736bd13538bcd58715421d"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.240185 7558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "5cc8f004a331463a887b2bff89ed6dc6"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "902e56a2d1a923707d88e262c8037f0c"
server_key_iv: "edfd3c65e1736bd13538bcd58715421d"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.242005 7558 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:23.242830 7573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.243027 7558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:23.243188 7558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "5cc8f004a331463a887b2bff89ed6dc6"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "902e56a2d1a923707d88e262c8037f0c"
server_key_iv: "edfd3c65e1736bd13538bcd58715421d"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.243355 7558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:23.261328 7558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:23.263314 7558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:23.263554 7558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:23.263819 7558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:23.264142 7558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:23.264196 7558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.264240 7558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:23.264272 7558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.270335 7558 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:37581
I20260923 22:17:23.270383 7686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:37581 every 8 connection(s)
I20260923 22:17:23.270717 7558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:23.278139 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 7558
I20260923 22:17:23.278228 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:23.278440 4924 external_mini_cluster.cc:1428] Setting key ba047c88fb83095a57a2c848e2295526
I20260923 22:17:23.280826 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:23.271467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:38557 (local address 127.4.207.62:43497)
0923 22:17:23.271529 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:23.271532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:23.271726 (+ 194us) server_negotiation.cc:407] Connection header received
0923 22:17:23.272190 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:23.272191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:23.272201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:23.272256 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:23.273207 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.273711 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.274852 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.275103 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.275161 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:23.277369 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:23.277371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:23.277371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:23.277377 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:23.279159 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:23.279744 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:23.279745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:23.279746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:23.279803 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:23.280174 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:23.280175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:23.280175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:23.280290 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:23.280345 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:23.280689 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 22:17:23.280760 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 22:17:23.281189 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:23.271388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43497 (local address 127.4.207.2:38557)
0923 22:17:23.271584 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:23.271628 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:23.272107 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:23.272312 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:23.272316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:23.272903 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:23.273115 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:23.273120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.273822 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.273823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:23.274772 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:23.274775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.275197 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.275198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:23.275307 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:23.275897 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:23.275905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:23.277273 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:23.279271 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:23.279279 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:23.279282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:23.279649 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:23.279877 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:23.279878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:23.279879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:23.280036 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:23.280351 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:23.280358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:23.280540 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 22:17:23.280653 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 22:17:23.280841 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":121,"threads_started":1}
I20260923 22:17:23.281855 7687 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43497
I20260923 22:17:23.281973 7687 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:23.282186 7687 heartbeater.cc:511] Master 127.4.207.62:43497 requested a full tablet report, sending...
I20260923 22:17:23.282686 7362 ts_manager.cc:194] Registered new tserver with Master: 5cc8f004a331463a887b2bff89ed6dc6 (127.4.207.2:37581)
I20260923 22:17:23.283093 7362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:38557
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:23.306473 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43497
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:23.388075 7692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:23.388329 7692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:23.388370 7692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:23.389580 7692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:23.389653 7692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:23.389700 7692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:23.391215 7692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:23.391533 7692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:23.391774 7692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:23.396701 7700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:23.396714 7697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:23.396796 7698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:23.396983 7692 server_base.cc:1034] running on GCE node
I20260923 22:17:23.397221 7692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:23.397462 7692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:23.398646 7692 hybrid_clock.cc:648] HybridClock initialized: now 1790201843398634 us; error 30 us; skew 500 ppm
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:23.401273 7692 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:23.401710 7692 webserver.cc:460] Webserver started at http://127.4.207.3:38681/ using document root <none> and password file <none>
I20260923 22:17:23.401896 7692 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:23.401935 7692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:23.402024 7692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:23.403118 7692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "8e32e496a66e4410a0df246e0ff08177"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "36959539b2ca9b269c4fce21626f0eae"
server_key_iv: "0cdb4ca4b08b5321c6802d31f78d51c4"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.403474 7692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "8e32e496a66e4410a0df246e0ff08177"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "36959539b2ca9b269c4fce21626f0eae"
server_key_iv: "0cdb4ca4b08b5321c6802d31f78d51c4"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.404762 7692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:23.405435 7707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.405597 7692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:23.405687 7692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "8e32e496a66e4410a0df246e0ff08177"
format_stamp: "Formatted at 2026-09-23 22:17:23 on dist-test-slave-7511"
server_key: "36959539b2ca9b269c4fce21626f0eae"
server_key_iv: "0cdb4ca4b08b5321c6802d31f78d51c4"
server_key_version: "encryptionkey@0"
I20260923 22:17:23.405742 7692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:23.419009 7692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:23.421319 7692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:23.421437 7692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:23.421633 7692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:23.421955 7692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:23.421990 7692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.422044 7692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:23.422081 7692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:23.427951 7692 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:40523
I20260923 22:17:23.428037 7820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:40523 every 8 connection(s)
I20260923 22:17:23.428404 7692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:23.431355 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 7692
I20260923 22:17:23.431505 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:23.431692 4924 external_mini_cluster.cc:1428] Setting key 1cbfbf1398e0b10cb665e40b48452484
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:23.438324 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:23.429231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:35373 (local address 127.4.207.62:43497)
0923 22:17:23.429272 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:23.429274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:23.429527 (+ 253us) server_negotiation.cc:407] Connection header received
0923 22:17:23.430074 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:23.430075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:23.430087 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:23.430140 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:23.431216 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.431916 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.433057 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.433268 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.433355 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:23.435532 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:23.435534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:23.435534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:23.435542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:23.436872 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:23.437310 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:23.437311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:23.437312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:23.437372 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:23.437628 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:23.437629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:23.437629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:23.437753 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:23.437839 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:23.438122 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 22:17:23.438232 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:17:23.438522 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:23.429233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43497 (local address 127.4.207.3:35373)
0923 22:17:23.429401 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:23.429437 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:23.429956 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:23.430220 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:23.430224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:23.430884 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:23.431087 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:23.431092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.431994 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.431995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:23.432957 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:23.432959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:23.433363 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:23.433364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:23.433489 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:23.433966 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:23.433974 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:23.435381 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:23.436988 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:23.436995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:23.436998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:23.437229 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:23.437440 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:23.437441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:23.437441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:23.437553 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:23.437829 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:23.437835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:23.437986 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 22:17:23.438102 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 22:17:23.438268 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260923 22:17:23.439052 7821 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43497
I20260923 22:17:23.439141 7821 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:23.439344 7821 heartbeater.cc:511] Master 127.4.207.62:43497 requested a full tablet report, sending...
I20260923 22:17:23.439769 7362 ts_manager.cc:194] Registered new tserver with Master: 8e32e496a66e4410a0df246e0ff08177 (127.4.207.3:40523)
I20260923 22:17:23.440163 7362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:35373
I20260923 22:17:23.443429 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 22:17:23.454499 7401 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:23.453514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.4.207.62:43497)
0923 22:17:23.453585 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:23.453588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:23.453715 (+ 127us) server_negotiation.cc:407] Connection header received
0923 22:17:23.453795 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:23.453796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:23.453807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:23.453882 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:23.454443 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36144: BlockingRecv error: recv got EOF from 127.0.0.1:36144 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:17:23.455924 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 7424
I20260923 22:17:23.460448 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 7558
I20260923 22:17:23.464598 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 7692
I20260923 22:17:23.468847 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 7330
2026-09-23T22:17:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1856 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7833](info): set up 2 sockets
Sep 23 22:17:23 dist-test-slave-7511 krb5kdc[7833](info): commencing operation
krb5kdc: starting...
W20260923 22:17:24.499261 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:24 dist-test-slave-7511 krb5kdc[7833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:24.618045 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:34381
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:34381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:24.702006 7846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:24.702244 7846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:24.702268 7846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:24.703632 7846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:24.703677 7846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:24.703692 7846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:24.703706 7846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:24.703717 7846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:24.705262 7846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:34381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:24.705528 7846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:24.705775 7846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:24.711000 7851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:24.710979 7854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.711009 7846 server_base.cc:1034] running on GCE node
W20260923 22:17:24.710979 7852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:24.711582 7846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:24.711884 7846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:24.713040 7846 hybrid_clock.cc:648] HybridClock initialized: now 1790201844713013 us; error 42 us; skew 500 ppm
Sep 23 22:17:24 dist-test-slave-7511 krb5kdc[7833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:24.715692 7846 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:24.716207 7846 webserver.cc:460] Webserver started at http://127.4.207.62:37597/ using document root <none> and password file <none>
I20260923 22:17:24.716465 7846 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:24.716535 7846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:24.716699 7846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:24.717835 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "bef777aa51a14be5ab0bd830ccfdaa04"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-7511"
server_key: "53a2b4a081b7b1e5c276f604cecb7dd3"
server_key_iv: "175d2a40492489458d423c05ee07a63e"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.718230 7846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "bef777aa51a14be5ab0bd830ccfdaa04"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-7511"
server_key: "53a2b4a081b7b1e5c276f604cecb7dd3"
server_key_iv: "175d2a40492489458d423c05ee07a63e"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.719743 7846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 22:17:24.720607 7861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:24.720903 7846 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:24.721022 7846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "bef777aa51a14be5ab0bd830ccfdaa04"
format_stamp: "Formatted at 2026-09-23 22:17:24 on dist-test-slave-7511"
server_key: "53a2b4a081b7b1e5c276f604cecb7dd3"
server_key_iv: "175d2a40492489458d423c05ee07a63e"
server_key_version: "encryptionkey@0"
I20260923 22:17:24.721104 7846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:24.733964 7846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:24.736055 7846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:24.736233 7846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:24.740159 7846 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:34381
I20260923 22:17:24.740195 7913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:34381 every 8 connection(s)
I20260923 22:17:24.740576 7846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:24.741482 7914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:24.743275 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 7846
I20260923 22:17:24.743369 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:24.743554 4924 external_mini_cluster.cc:1428] Setting key 79889e8aab9d9bcfe85cdc2ee4e157f9
I20260923 22:17:24.744266 7914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04: Bootstrap starting.
I20260923 22:17:24.744933 7914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:24.745304 7914 log.cc:824] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:24.746068 7914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04: No bootstrap required, opened a new log
I20260923 22:17:24.748309 7914 raft_consensus.cc:348] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef777aa51a14be5ab0bd830ccfdaa04" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34381 } }
I20260923 22:17:24.748490 7914 raft_consensus.cc:374] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:24.748539 7914 raft_consensus.cc:729] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef777aa51a14be5ab0bd830ccfdaa04, State: Initialized, Role: FOLLOWER
I20260923 22:17:24.748687 7914 consensus_queue.cc:260] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef777aa51a14be5ab0bd830ccfdaa04" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34381 } }
I20260923 22:17:24.748787 7914 raft_consensus.cc:388] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:24.748813 7914 raft_consensus.cc:482] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:24.748883 7914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:24.749714 7914 raft_consensus.cc:504] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef777aa51a14be5ab0bd830ccfdaa04" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34381 } }
I20260923 22:17:24.749845 7914 leader_election.cc:302] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bef777aa51a14be5ab0bd830ccfdaa04; no voters:
I20260923 22:17:24.750025 7914 leader_election.cc:288] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:24.750092 7919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:24.750262 7919 raft_consensus.cc:686] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [term 1 LEADER]: Becoming Leader. State: Replica: bef777aa51a14be5ab0bd830ccfdaa04, State: Running, Role: LEADER
I20260923 22:17:24.750308 7914 sys_catalog.cc:564] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:24.750389 7919 consensus_queue.cc:237] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef777aa51a14be5ab0bd830ccfdaa04" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34381 } }
I20260923 22:17:24.750715 7919 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bef777aa51a14be5ab0bd830ccfdaa04. Latest consensus state: current_term: 1 leader_uuid: "bef777aa51a14be5ab0bd830ccfdaa04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef777aa51a14be5ab0bd830ccfdaa04" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34381 } } }
I20260923 22:17:24.750808 7919 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [sys.catalog]: This master's current role is: LEADER
Sep 23 22:17:24 dist-test-slave-7511 krb5kdc[7833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:24.750730 7920 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bef777aa51a14be5ab0bd830ccfdaa04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef777aa51a14be5ab0bd830ccfdaa04" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34381 } } }
I20260923 22:17:24.750962 7920 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:24.751078 7926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:24.751645 7926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:24.753376 7926 catalog_manager.cc:1347] Generated new cluster ID: 1a35b9a220934290ab9dc14c212965a3
I20260923 22:17:24.753430 7926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:24.755292 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.744100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.4.207.62:34381)
0923 22:17:24.744298 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.744307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.744356 (+ 49us) server_negotiation.cc:407] Connection header received
0923 22:17:24.744810 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.744817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.744865 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.744986 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:24.745841 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.746769 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.747977 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.748382 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.748459 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.751513 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:24.751518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:24.751519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:24.751534 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:24.753652 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:24.754072 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:24.754075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:24.754076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:24.754183 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:24.754387 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:24.754388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:24.754389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:24.754699 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:24.754861 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.754939 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.755094 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260923 22:17:24.756285 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:24.773102 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:24.768634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.4.207.62:34381)
0923 22:17:24.768682 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:24.768684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:24.768819 (+ 135us) server_negotiation.cc:407] Connection header received
0923 22:17:24.768892 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:24.768893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:24.768905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:24.768978 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:24.769469 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.770218 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.771306 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:24.771656 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:24.771723 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:24.772536 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:24.772538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:24.772538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:24.772810 (+ 272us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:24.772830 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:24.772837 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:24.772897 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:24.772980 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 22:17:24.773009 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:24.773823 7926 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:24.774344 7926 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:24.784618 7926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bef777aa51a14be5ab0bd830ccfdaa04: Generated new TSK 0
I20260923 22:17:24.784860 7926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:24.788340 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 7846
2026-09-23T22:17:24Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1317 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:24 dist-test-slave-7511 krb5kdc[7947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:24 dist-test-slave-7511 krb5kdc[7947](info): set up 2 sockets
Sep 23 22:17:24 dist-test-slave-7511 krb5kdc[7947](info): commencing operation
krb5kdc: starting...
W20260923 22:17:25.823338 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:25 dist-test-slave-7511 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:25.943107 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:41465
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:41465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:26.024109 7960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:26.024374 7960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:26.024415 7960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:26.025621 7960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:26.025699 7960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:26.025723 7960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:26.025745 7960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:26.025774 7960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:26.027292 7960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:41465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:41465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:26.027603 7960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:26.027861 7960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:26.032966 7966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:26.032958 7968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:26.032995 7960 server_base.cc:1034] running on GCE node
W20260923 22:17:26.032959 7965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:26.033383 7960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:26.033710 7960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:26.034881 7960 hybrid_clock.cc:648] HybridClock initialized: now 1790201846034870 us; error 47 us; skew 500 ppm
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:26.037474 7960 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:26.037945 7960 webserver.cc:460] Webserver started at http://127.4.207.62:41331/ using document root <none> and password file <none>
I20260923 22:17:26.038187 7960 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:26.038255 7960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:26.038403 7960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:26.039553 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "446d92a96b674a399934bcd16328cb0a"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "4dbe5251cbf7a14728c5e6fa58f90122"
server_key_iv: "339f20e16adbdf518719b26001f6dee4"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.039933 7960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "446d92a96b674a399934bcd16328cb0a"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "4dbe5251cbf7a14728c5e6fa58f90122"
server_key_iv: "339f20e16adbdf518719b26001f6dee4"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.041225 7960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:26.041942 7975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.042110 7960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.042218 7960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "446d92a96b674a399934bcd16328cb0a"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "4dbe5251cbf7a14728c5e6fa58f90122"
server_key_iv: "339f20e16adbdf518719b26001f6dee4"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.042292 7960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:26.063289 7960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:26.065848 7960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:26.065977 7960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:26.069864 7960 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:41465
I20260923 22:17:26.069933 8027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:41465 every 8 connection(s)
I20260923 22:17:26.070277 7960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:26.071316 8028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:26.073722 8028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a: Bootstrap starting.
I20260923 22:17:26.074297 8028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:26.074571 8028 log.cc:824] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:26.075327 8028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a: No bootstrap required, opened a new log
I20260923 22:17:26.076975 8028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d92a96b674a399934bcd16328cb0a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41465 } }
I20260923 22:17:26.077147 8028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:26.077196 8028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446d92a96b674a399934bcd16328cb0a, State: Initialized, Role: FOLLOWER
I20260923 22:17:26.077373 8028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d92a96b674a399934bcd16328cb0a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41465 } }
I20260923 22:17:26.077477 8028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:26.077520 8028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:26.077569 8028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:26.077939 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 7960
I20260923 22:17:26.078024 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:26.078223 4924 external_mini_cluster.cc:1428] Setting key 6794787be1dd8b6d02efccd072d32b08
I20260923 22:17:26.078593 8028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d92a96b674a399934bcd16328cb0a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41465 } }
I20260923 22:17:26.078734 8028 leader_election.cc:302] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 446d92a96b674a399934bcd16328cb0a; no voters:
I20260923 22:17:26.078903 8028 leader_election.cc:288] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:26.078948 8032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:26.079069 8032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [term 1 LEADER]: Becoming Leader. State: Replica: 446d92a96b674a399934bcd16328cb0a, State: Running, Role: LEADER
I20260923 22:17:26.079169 8028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:26.079264 8032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d92a96b674a399934bcd16328cb0a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41465 } }
I20260923 22:17:26.079658 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446d92a96b674a399934bcd16328cb0a. Latest consensus state: current_term: 1 leader_uuid: "446d92a96b674a399934bcd16328cb0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d92a96b674a399934bcd16328cb0a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41465 } } }
I20260923 22:17:26.079892 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:26.079820 8034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "446d92a96b674a399934bcd16328cb0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446d92a96b674a399934bcd16328cb0a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41465 } } }
I20260923 22:17:26.080367 8034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:26.081032 8049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:26.081104 8049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:26.081171 8042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:26.081341 8042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:26.083267 8042 catalog_manager.cc:1347] Generated new cluster ID: 0eb8f44f45ae4ed5bfbf207d453921d1
I20260923 22:17:26.083338 8042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:26.090579 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.078801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.4.207.62:41465)
0923 22:17:26.079102 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:26.079114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:26.079152 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:17:26.079306 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:26.079314 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:26.079378 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:26.080428 (+ 1050us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:26.080539 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.082058 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.082947 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.083632 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.083746 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.086522 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:26.086526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:26.086527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:26.086540 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:26.088830 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.089218 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.089223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.089223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.089367 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.089587 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.089588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.089588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.089933 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:26.090090 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:26.090179 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 22:17:26.090321 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
I20260923 22:17:26.105255 8042 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:26.105808 8042 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:26.110288 8042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 446d92a96b674a399934bcd16328cb0a: Generated new TSK 0
I20260923 22:17:26.110509 8042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:26.142908 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41465
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:26.243431 8054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:26.243708 8054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:26.243747 8054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:26.245003 8054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:26.245086 8054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:26.245132 8054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:26.246915 8054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:26.247232 8054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:26.247519 8054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:26.252743 8059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:26.252797 8062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:26.252800 8060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:26.252961 8054 server_base.cc:1034] running on GCE node
I20260923 22:17:26.253199 8054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:26.253482 8054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:26.254623 8054 hybrid_clock.cc:648] HybridClock initialized: now 1790201846254598 us; error 38 us; skew 500 ppm
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:26.257153 8054 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:26.257650 8054 webserver.cc:460] Webserver started at http://127.4.207.1:37557/ using document root <none> and password file <none>
I20260923 22:17:26.257887 8054 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:26.257961 8054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:26.258152 8054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:26.259256 8054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "687f1469a83c4356b473ab67a6ddd642"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "beb0c2e654d4a7be7b3e0474eb285e20"
server_key_iv: "81a6c9f2c43e8e9b012acb57cf781d7f"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.259645 8054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "687f1469a83c4356b473ab67a6ddd642"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "beb0c2e654d4a7be7b3e0474eb285e20"
server_key_iv: "81a6c9f2c43e8e9b012acb57cf781d7f"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.260980 8054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:26.261739 8069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.261911 8054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:26.262040 8054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "687f1469a83c4356b473ab67a6ddd642"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "beb0c2e654d4a7be7b3e0474eb285e20"
server_key_iv: "81a6c9f2c43e8e9b012acb57cf781d7f"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.262117 8054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:26.283743 8054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:26.285782 8054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:26.285964 8054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:26.286239 8054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:26.286594 8054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:26.286652 8054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.286693 8054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:26.286722 8054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.292632 8054 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:41687
I20260923 22:17:26.292676 8182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:41687 every 8 connection(s)
I20260923 22:17:26.293063 8054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:26.298918 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 8054
I20260923 22:17:26.299010 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:26.299239 4924 external_mini_cluster.cc:1428] Setting key 949ae8cc7efe8d9451142e5ec102740a
I20260923 22:17:26.302901 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.293852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:54915 (local address 127.4.207.62:41465)
0923 22:17:26.293936 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:26.293938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:26.294184 (+ 246us) server_negotiation.cc:407] Connection header received
0923 22:17:26.294671 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:26.294672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:26.294685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:26.294742 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:26.295716 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.296200 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.297171 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.297362 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.297430 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.299710 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:26.299711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:26.299712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:26.299720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:26.301374 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.301921 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.301922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.301923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.301975 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.302259 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.302260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.302260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.302361 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:26.302410 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:26.302759 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 22:17:26.302837 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 22:17:26.303189 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.293831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41465 (local address 127.4.207.1:54915)
0923 22:17:26.293998 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:26.294095 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:26.294584 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:26.294798 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:26.294801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:26.295398 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:26.295627 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:26.295633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.296295 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.296295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:26.297098 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:26.297100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.297456 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.297456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:26.297575 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.298069 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:26.298077 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:26.299630 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:26.301470 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:26.301479 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:26.301481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:26.301841 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:26.302043 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:26.302044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:26.302044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:26.302194 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:26.302428 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:26.302434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:26.302622 (+ 188us) client_negotiation.cc:769] Sending connection context
0923 22:17:26.302739 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 22:17:26.302931 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":105,"threads_started":1}
I20260923 22:17:26.303799 8183 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41465
I20260923 22:17:26.303925 8183 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:26.304136 8183 heartbeater.cc:511] Master 127.4.207.62:41465 requested a full tablet report, sending...
I20260923 22:17:26.304672 7992 ts_manager.cc:194] Registered new tserver with Master: 687f1469a83c4356b473ab67a6ddd642 (127.4.207.1:41687)
I20260923 22:17:26.305719 7992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:54915
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:26.326946 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41465
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:26.408073 8188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:26.408409 8188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:26.408463 8188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:26.409668 8188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:26.409718 8188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:26.409744 8188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:26.411365 8188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:26.411643 8188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:26.411880 8188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:26.417104 8194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:26.417301 8188 server_base.cc:1034] running on GCE node
W20260923 22:17:26.417099 8196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:26.417189 8193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:26.417644 8188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:26.417922 8188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:26.419126 8188 hybrid_clock.cc:648] HybridClock initialized: now 1790201846419102 us; error 38 us; skew 500 ppm
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:26.421686 8188 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:26.422178 8188 webserver.cc:460] Webserver started at http://127.4.207.2:42187/ using document root <none> and password file <none>
I20260923 22:17:26.422376 8188 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:26.422420 8188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:26.422513 8188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:26.423676 8188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "1cdfcece56d244e1b6597f65da6319fc"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "2625a41a19ff3ca9e01f7faf7b8a6c2d"
server_key_iv: "dd8fa0d2e0c0213c37f446aa84c2d924"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.423988 8188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "1cdfcece56d244e1b6597f65da6319fc"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "2625a41a19ff3ca9e01f7faf7b8a6c2d"
server_key_iv: "dd8fa0d2e0c0213c37f446aa84c2d924"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.425364 8188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:26.426229 8203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.426424 8188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:26.426503 8188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "1cdfcece56d244e1b6597f65da6319fc"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "2625a41a19ff3ca9e01f7faf7b8a6c2d"
server_key_iv: "dd8fa0d2e0c0213c37f446aa84c2d924"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.426564 8188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:26.447871 8188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:26.450204 8188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:26.450314 8188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:26.450517 8188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:26.450775 8188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:26.450804 8188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.450827 8188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:26.450839 8188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.456589 8188 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:36825
I20260923 22:17:26.456665 8316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:36825 every 8 connection(s)
I20260923 22:17:26.456979 8188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:26.462098 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 8188
I20260923 22:17:26.462222 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:26.462411 4924 external_mini_cluster.cc:1428] Setting key 0c0f8e3033d51683ca35558551a04607
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:26.467810 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.457747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:50001 (local address 127.4.207.62:41465)
0923 22:17:26.457819 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:26.457823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:26.458067 (+ 244us) server_negotiation.cc:407] Connection header received
0923 22:17:26.458556 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:26.458557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:26.458567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:26.458621 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:26.459615 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.460112 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.461143 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.461341 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.461413 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.464343 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:26.464344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:26.464344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:26.464352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:26.466204 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.466717 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.466717 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.466718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.466772 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.467093 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.467094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.467094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.467200 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:26.467250 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:26.467657 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 22:17:26.467730 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:26.468112 8319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.457744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41465 (local address 127.4.207.2:50001)
0923 22:17:26.457936 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:26.457977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:26.458463 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:26.458678 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:26.458681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:26.459269 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:26.459524 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:26.459537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.460188 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.460189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:26.461012 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:26.461013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.461426 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.461427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:26.461540 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.462181 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:26.462190 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:26.464219 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:26.466285 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:26.466293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:26.466296 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:26.466641 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:26.466841 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:26.466842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:26.466843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:26.466994 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:26.467272 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:26.467279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:26.467496 (+ 217us) client_negotiation.cc:769] Sending connection context
0923 22:17:26.467622 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 22:17:26.467834 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260923 22:17:26.468673 8317 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41465
I20260923 22:17:26.468801 8317 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:26.468989 8317 heartbeater.cc:511] Master 127.4.207.62:41465 requested a full tablet report, sending...
I20260923 22:17:26.469422 7992 ts_manager.cc:194] Registered new tserver with Master: 1cdfcece56d244e1b6597f65da6319fc (127.4.207.2:36825)
I20260923 22:17:26.469805 7992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:50001
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:26.490384 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41465
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:26.571314 8322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:26.571585 8322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:26.571630 8322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:26.572798 8322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:26.572866 8322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:26.572909 8322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:26.574319 8322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41465
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:26.574585 8322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:26.574819 8322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:26.579775 8330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:26.579836 8327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:26.579799 8328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:26.580191 8322 server_base.cc:1034] running on GCE node
I20260923 22:17:26.580454 8322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:26.580749 8322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:26.581905 8322 hybrid_clock.cc:648] HybridClock initialized: now 1790201846581875 us; error 48 us; skew 500 ppm
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:26.584528 8322 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:26.584951 8322 webserver.cc:460] Webserver started at http://127.4.207.3:39183/ using document root <none> and password file <none>
I20260923 22:17:26.585155 8322 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:26.585194 8322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:26.585284 8322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:26.586262 8322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "6ee14f3fdc324cd7b870145efeabc78d"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "2992a186b3d11558f398917f305986ec"
server_key_iv: "08f1aa55d784127ca519c45003c61f9d"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.586566 8322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "6ee14f3fdc324cd7b870145efeabc78d"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "2992a186b3d11558f398917f305986ec"
server_key_iv: "08f1aa55d784127ca519c45003c61f9d"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.587883 8322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 22:17:26.588552 8337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.588714 8322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:26.588781 8322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "6ee14f3fdc324cd7b870145efeabc78d"
format_stamp: "Formatted at 2026-09-23 22:17:26 on dist-test-slave-7511"
server_key: "2992a186b3d11558f398917f305986ec"
server_key_iv: "08f1aa55d784127ca519c45003c61f9d"
server_key_version: "encryptionkey@0"
I20260923 22:17:26.588835 8322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:26.608309 8322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:26.610872 8322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:26.610992 8322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:26.611177 8322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:26.611500 8322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:26.611544 8322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.611567 8322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:26.611580 8322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:26.617503 8322 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:34719
I20260923 22:17:26.617547 8450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:34719 every 8 connection(s)
I20260923 22:17:26.617874 8322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:26.625516 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 8322
I20260923 22:17:26.625651 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:26.625821 4924 external_mini_cluster.cc:1428] Setting key 03b88bac99fb3f72d9b2bb551a73acc6
I20260923 22:17:26.627363 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.618624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:55317 (local address 127.4.207.62:41465)
0923 22:17:26.618711 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:26.618714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:26.618886 (+ 172us) server_negotiation.cc:407] Connection header received
0923 22:17:26.619392 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:26.619393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:26.619404 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:26.619523 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:26.620487 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.620973 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.621955 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.622146 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.622196 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.624389 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:26.624390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:26.624391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:26.624398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:26.625932 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.626371 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.626372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.626373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.626439 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.626703 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.626704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.626704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.626828 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:26.626908 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:26.627189 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 22:17:26.627270 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 22:17:26.627615 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.618576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41465 (local address 127.4.207.3:55317)
0923 22:17:26.618754 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:26.618791 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:26.619303 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:26.619598 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:26.619602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:26.620195 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:26.620392 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:26.620396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.621050 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.621051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:26.621876 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:26.621878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.622217 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.622217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:26.622317 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.622781 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:26.622790 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:26.624275 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:26.626041 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:26.626049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:26.626052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:26.626292 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:26.626515 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:26.626515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:26.626516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:26.626633 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:26.626897 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:26.626903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:26.627053 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 22:17:26.627167 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 22:17:26.627331 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260923 22:17:26.628149 8451 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41465
I20260923 22:17:26.628250 8451 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:26.628392 8451 heartbeater.cc:511] Master 127.4.207.62:41465 requested a full tablet report, sending...
I20260923 22:17:26.628754 7992 ts_manager.cc:194] Registered new tserver with Master: 6ee14f3fdc324cd7b870145efeabc78d (127.4.207.3:34719)
I20260923 22:17:26.629108 7992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:55317
I20260923 22:17:26.637643 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[7947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201846, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:26.661082 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:26.653798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.4.207.62:41465)
0923 22:17:26.653874 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:26.653876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:26.653961 (+ 85us) server_negotiation.cc:407] Connection header received
0923 22:17:26.654009 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:26.654010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:26.654020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:26.654108 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:26.654673 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.655180 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.656002 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:26.656207 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:26.656270 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:26.658483 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:26.658484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:26.658484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:26.658491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:26.659986 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.660319 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.660319 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.660320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.660369 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:26.660554 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:26.660554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:26.660555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:26.660698 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:26.660773 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:26.660926 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 22:17:26.660987 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 22:17:26.661314 7992 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41186
I20260923 22:17:26.661937 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 8054
I20260923 22:17:26.666330 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 8188
I20260923 22:17:26.670504 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 8322
I20260923 22:17:26.674733 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 7960
2026-09-23T22:17:26Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1888 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[8463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[8463](info): set up 2 sockets
Sep 23 22:17:26 dist-test-slave-7511 krb5kdc[8463](info): commencing operation
krb5kdc: starting...
W20260923 22:17:27.715595 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:27 dist-test-slave-7511 krb5kdc[8463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:27.834488 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:43043
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:43043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:27.916371 8476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:27.916594 8476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:27.916617 8476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:27.917729 8476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:27.917761 8476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:27.917774 8476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:27.917788 8476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:27.917798 8476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:27.919262 8476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:43043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:27.919533 8476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:27.919741 8476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:27.924620 8482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:27.924651 8481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:27.924620 8484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:27.924849 8476 server_base.cc:1034] running on GCE node
I20260923 22:17:27.925103 8476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:27.925344 8476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:27.926507 8476 hybrid_clock.cc:648] HybridClock initialized: now 1790201847926490 us; error 32 us; skew 500 ppm
Sep 23 22:17:27 dist-test-slave-7511 krb5kdc[8463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:27.929126 8476 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:27.929622 8476 webserver.cc:460] Webserver started at http://127.4.207.62:46853/ using document root <none> and password file <none>
I20260923 22:17:27.929859 8476 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:27.929924 8476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:27.930071 8476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:27.931131 8476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "10ae1b8352a840c98c369bfff7c39dad"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-7511"
server_key: "eb6541f6fcef0bc3ecd620f7482494eb"
server_key_iv: "9e019836c629f2f3ea1ac1dbf1200551"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.931591 8476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "10ae1b8352a840c98c369bfff7c39dad"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-7511"
server_key: "eb6541f6fcef0bc3ecd620f7482494eb"
server_key_iv: "9e019836c629f2f3ea1ac1dbf1200551"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.932934 8476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:27.933692 8491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:27.933961 8476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:27.934085 8476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "10ae1b8352a840c98c369bfff7c39dad"
format_stamp: "Formatted at 2026-09-23 22:17:27 on dist-test-slave-7511"
server_key: "eb6541f6fcef0bc3ecd620f7482494eb"
server_key_iv: "9e019836c629f2f3ea1ac1dbf1200551"
server_key_version: "encryptionkey@0"
I20260923 22:17:27.934171 8476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:27.961802 8476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:27.964295 8476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:27.964468 8476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:27.968514 8476 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:43043
I20260923 22:17:27.968555 8543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:43043 every 8 connection(s)
I20260923 22:17:27.969091 8476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:27.969808 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 8476
I20260923 22:17:27.969834 8544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:27.969920 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:27.970103 4924 external_mini_cluster.cc:1428] Setting key c14f6bdcd6c521e9c6fc0add620ebec1
I20260923 22:17:27.973027 8544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad: Bootstrap starting.
I20260923 22:17:27.973793 8544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:27.974085 8544 log.cc:824] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:27.974845 8544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad: No bootstrap required, opened a new log
Sep 23 22:17:27 dist-test-slave-7511 krb5kdc[8463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:27.977560 8544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae1b8352a840c98c369bfff7c39dad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43043 } }
I20260923 22:17:27.977723 8544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:27.977771 8544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ae1b8352a840c98c369bfff7c39dad, State: Initialized, Role: FOLLOWER
I20260923 22:17:27.977936 8544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae1b8352a840c98c369bfff7c39dad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43043 } }
I20260923 22:17:27.978034 8544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:27.978084 8544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:27.978132 8544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:27.978662 8544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae1b8352a840c98c369bfff7c39dad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43043 } }
I20260923 22:17:27.978789 8544 leader_election.cc:302] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10ae1b8352a840c98c369bfff7c39dad; no voters:
I20260923 22:17:27.978992 8544 leader_election.cc:288] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:27.979022 8549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:27.979127 8549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [term 1 LEADER]: Becoming Leader. State: Replica: 10ae1b8352a840c98c369bfff7c39dad, State: Running, Role: LEADER
I20260923 22:17:27.979208 8549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae1b8352a840c98c369bfff7c39dad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43043 } }
I20260923 22:17:27.979271 8544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:27.979558 8549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10ae1b8352a840c98c369bfff7c39dad. Latest consensus state: current_term: 1 leader_uuid: "10ae1b8352a840c98c369bfff7c39dad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae1b8352a840c98c369bfff7c39dad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43043 } } }
I20260923 22:17:27.979702 8549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:27.979606 8551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10ae1b8352a840c98c369bfff7c39dad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ae1b8352a840c98c369bfff7c39dad" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43043 } } }
I20260923 22:17:27.979921 8551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:27.981091 8564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:27.981184 8564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:27.981292 8554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:27.981460 8554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:27.981503 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:27.970627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.4.207.62:43043)
0923 22:17:27.970805 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:27.970814 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:27.970853 (+ 39us) server_negotiation.cc:407] Connection header received
0923 22:17:27.971312 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:27.971318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:27.971385 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:27.971571 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:27.972302 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.973237 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.974315 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:27.974831 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:27.974939 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:27.977564 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:27.977567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:27.977568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:27.977578 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:27.979758 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:27.980172 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:27.980175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:27.980175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:27.980273 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:27.980487 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:27.980488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:27.980488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:27.980841 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:27.980997 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:27.981067 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 22:17:27.981300 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260923 22:17:27.982954 8554 catalog_manager.cc:1347] Generated new cluster ID: b127a07da97e4258bc1fcb6b4b089f44
I20260923 22:17:27.982998 8554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:28.035781 8554 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:28.036376 8554 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:28.040424 8554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10ae1b8352a840c98c369bfff7c39dad: Generated new TSK 0
I20260923 22:17:28.040601 8554 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:28.067667 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43043
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:28.167296 8570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:28.167532 8570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:28.167557 8570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:28.168751 8570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:28.168798 8570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:28.168825 8570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:28.170394 8570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:28.170660 8570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:28.170899 8570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:28.175935 8576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:28.175973 8575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:28.175932 8578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:28.176256 8570 server_base.cc:1034] running on GCE node
I20260923 22:17:28.176441 8570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:28.176638 8570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:28.177781 8570 hybrid_clock.cc:648] HybridClock initialized: now 1790201848177769 us; error 26 us; skew 500 ppm
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.180339 8570 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:28.180809 8570 webserver.cc:460] Webserver started at http://127.4.207.1:44343/ using document root <none> and password file <none>
I20260923 22:17:28.181066 8570 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:28.181113 8570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:28.181205 8570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:28.182211 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "5bf72000e75a4e6f99b97ef4a85bb9ba"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "66d11eb678b81029635af5a2c7fc5cbd"
server_key_iv: "7cc90c5853443bcc4ff0bd34c2e2ffc2"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.182505 8570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "5bf72000e75a4e6f99b97ef4a85bb9ba"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "66d11eb678b81029635af5a2c7fc5cbd"
server_key_iv: "7cc90c5853443bcc4ff0bd34c2e2ffc2"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.183775 8570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:28.184463 8585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.184643 8570 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:28.184731 8570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "5bf72000e75a4e6f99b97ef4a85bb9ba"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "66d11eb678b81029635af5a2c7fc5cbd"
server_key_iv: "7cc90c5853443bcc4ff0bd34c2e2ffc2"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.184804 8570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:28.202848 8570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:28.204967 8570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:28.205122 8570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:28.205374 8570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:28.205693 8570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:28.205725 8570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.205749 8570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:28.205793 8570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.211535 8570 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:46019
I20260923 22:17:28.211611 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:46019 every 8 connection(s)
I20260923 22:17:28.211941 8570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:28.213655 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 8570
I20260923 22:17:28.213745 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:28.213963 4924 external_mini_cluster.cc:1428] Setting key 4cfb349c52923a034970df88edd67697
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
I20260923 22:17:28.223217 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.212695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:44773 (local address 127.4.207.62:43043)
0923 22:17:28.212765 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.212767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.212989 (+ 222us) server_negotiation.cc:407] Connection header received
0923 22:17:28.214005 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.214007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.214020 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.214067 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.214947 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.215655 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.217196 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.217531 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.217612 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.220117 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.220118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.220119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.220126 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.221630 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.222134 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.222135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.222135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.222195 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.222488 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.222488 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.222489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.222608 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.222709 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.223061 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.223149 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:17:28.223343 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.212679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43043 (local address 127.4.207.1:44773)
0923 22:17:28.212861 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:28.212903 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:28.213896 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:28.214160 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:28.214166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:28.214568 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:28.214838 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:28.214844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.215749 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.215750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:28.217093 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:28.217096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.217637 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.217638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:28.217771 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.218323 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:28.218331 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:28.219996 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:28.221720 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:28.221726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:28.221729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:28.222049 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:28.222276 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:28.222277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:28.222277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:28.222422 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:28.222708 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:28.222716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:28.222887 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 22:17:28.223009 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 22:17:28.223147 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":108,"threads_started":1}
I20260923 22:17:28.223927 8699 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43043
I20260923 22:17:28.224031 8699 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:28.224268 8699 heartbeater.cc:511] Master 127.4.207.62:43043 requested a full tablet report, sending...
I20260923 22:17:28.224890 8508 ts_manager.cc:194] Registered new tserver with Master: 5bf72000e75a4e6f99b97ef4a85bb9ba (127.4.207.1:46019)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
I20260923 22:17:28.225716 8508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:44773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:28.241833 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43043
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:28.325862 8704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:28.326098 8704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:28.326125 8704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:28.327348 8704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:28.327404 8704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:28.327488 8704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:28.329146 8704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:28.329427 8704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:28.329710 8704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:28.335021 8709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:28.335103 8712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:28.335043 8710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:28.335199 8704 server_base.cc:1034] running on GCE node
I20260923 22:17:28.335544 8704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:28.335765 8704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:28.336933 8704 hybrid_clock.cc:648] HybridClock initialized: now 1790201848336925 us; error 42 us; skew 500 ppm
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.339612 8704 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:28.340055 8704 webserver.cc:460] Webserver started at http://127.4.207.2:38687/ using document root <none> and password file <none>
I20260923 22:17:28.340296 8704 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:28.340366 8704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:28.340514 8704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:28.341599 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "0ddb1b16bdad4881907d3b77a6986624"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "38be1c2d2b41d824d0a3e4b72d68ef46"
server_key_iv: "e2962642b7618512beb209ebf5825104"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.341961 8704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "0ddb1b16bdad4881907d3b77a6986624"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "38be1c2d2b41d824d0a3e4b72d68ef46"
server_key_iv: "e2962642b7618512beb209ebf5825104"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.343295 8704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:28.344102 8719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.344266 8704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:28.344364 8704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "0ddb1b16bdad4881907d3b77a6986624"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "38be1c2d2b41d824d0a3e4b72d68ef46"
server_key_iv: "e2962642b7618512beb209ebf5825104"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.344447 8704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:28.368861 8704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:28.370788 8704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:28.370963 8704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:28.371209 8704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:28.371557 8704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:28.371619 8704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.371656 8704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:28.371711 8704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.377612 8704 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:37177
I20260923 22:17:28.377651 8832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:37177 every 8 connection(s)
I20260923 22:17:28.377977 8704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:28.386976 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 8704
I20260923 22:17:28.387125 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:28.387414 4924 external_mini_cluster.cc:1428] Setting key 12943607016bf20efa89ce9d0742c56c
I20260923 22:17:28.387982 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.378752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:56841 (local address 127.4.207.62:43043)
0923 22:17:28.378823 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.378826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.379070 (+ 244us) server_negotiation.cc:407] Connection header received
0923 22:17:28.380246 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.380248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.380259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.380308 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.380990 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.381478 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.382444 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.382608 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.382654 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.384969 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.384971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.384971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.384979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.386213 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.386650 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.386651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.386651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.386710 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.387065 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.387067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.387067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.387267 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.387355 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.387801 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.387889 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:28.388118 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.378733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43043 (local address 127.4.207.2:56841)
0923 22:17:28.378920 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:28.378961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:28.380119 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:28.380396 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:28.380403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:28.380714 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:28.380914 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:28.380918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.381561 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.381562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:28.382375 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:28.382377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.382675 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.382676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:28.382792 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.383361 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:28.383375 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:28.384863 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:28.386299 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:28.386305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:28.386308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:28.386536 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:28.386777 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:28.386778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:28.386779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:28.386942 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:28.387364 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:28.387370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:28.387603 (+ 233us) client_negotiation.cc:769] Sending connection context
0923 22:17:28.387745 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 22:17:28.387914 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":102,"threads_started":1}
I20260923 22:17:28.388638 8833 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43043
I20260923 22:17:28.388757 8833 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:28.388936 8833 heartbeater.cc:511] Master 127.4.207.62:43043 requested a full tablet report, sending...
I20260923 22:17:28.389341 8508 ts_manager.cc:194] Registered new tserver with Master: 0ddb1b16bdad4881907d3b77a6986624 (127.4.207.2:37177)
I20260923 22:17:28.389725 8508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:56841
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:28.413920 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43043
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:38423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:28.496675 8838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:28.496923 8838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:28.496963 8838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:28.498139 8838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:28.498237 8838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:28.498281 8838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:28.499902 8838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:38423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43043
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:28.500183 8838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:28.500416 8838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:28.505332 8846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:28.505332 8843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:28.505612 8838 server_base.cc:1034] running on GCE node
W20260923 22:17:28.505339 8844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:28.505892 8838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:28.506141 8838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:28.507287 8838 hybrid_clock.cc:648] HybridClock initialized: now 1790201848507274 us; error 29 us; skew 500 ppm
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:28.510115 8838 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:28.510568 8838 webserver.cc:460] Webserver started at http://127.4.207.3:35751/ using document root <none> and password file <none>
I20260923 22:17:28.510864 8838 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:28.510936 8838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:28.511081 8838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:28.512229 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "c2fbb7fb4c134f6787589e521d0790cb"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "38626cbed6556bf56fa3cf0e9fbdfe84"
server_key_iv: "e8ae951517c176de3b3bcd290ba7e24d"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.512601 8838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "c2fbb7fb4c134f6787589e521d0790cb"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "38626cbed6556bf56fa3cf0e9fbdfe84"
server_key_iv: "e8ae951517c176de3b3bcd290ba7e24d"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.514240 8838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:28.515118 8853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.515304 8838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:28.515406 8838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "c2fbb7fb4c134f6787589e521d0790cb"
format_stamp: "Formatted at 2026-09-23 22:17:28 on dist-test-slave-7511"
server_key: "38626cbed6556bf56fa3cf0e9fbdfe84"
server_key_iv: "e8ae951517c176de3b3bcd290ba7e24d"
server_key_version: "encryptionkey@0"
I20260923 22:17:28.515499 8838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:28.526995 8838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:28.529259 8838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:28.529444 8838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:28.529721 8838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:28.530071 8838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:28.530131 8838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.530182 8838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:28.530218 8838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:28.536667 8838 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:40265
I20260923 22:17:28.536715 8966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:40265 every 8 connection(s)
I20260923 22:17:28.537092 8838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:28.539040 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 8838
I20260923 22:17:28.539173 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:28.539386 4924 external_mini_cluster.cc:1428] Setting key 12484694fc7f41df4589e524b597d4ae
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:28.547128 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.537863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:38577 (local address 127.4.207.62:43043)
0923 22:17:28.537928 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.537930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.538136 (+ 206us) server_negotiation.cc:407] Connection header received
0923 22:17:28.539126 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.539127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.539141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.539191 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.540110 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.540863 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.541897 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.542079 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.542139 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.544331 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.544332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.544333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.544340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.545744 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.546166 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.546167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.546167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.546234 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.546482 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.546483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.546484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.546602 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.546682 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.546956 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.547033 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:17:28.547331 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.537835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43043 (local address 127.4.207.3:38577)
0923 22:17:28.538007 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:28.538044 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:28.539001 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:28.539267 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:28.539270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:28.539657 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:28.540018 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:28.540030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.540957 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.540958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:28.541817 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:28.541820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.542157 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.542158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:28.542268 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.542757 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:28.542765 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:28.544223 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:28.545847 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:28.545855 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:28.545857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:28.546088 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:28.546303 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:28.546304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:28.546305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:28.546414 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:28.546672 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:28.546678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:28.546825 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 22:17:28.546936 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 22:17:28.547097 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":119,"threads_started":1}
I20260923 22:17:28.547873 8967 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43043
I20260923 22:17:28.547969 8967 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:28.548111 8967 heartbeater.cc:511] Master 127.4.207.62:43043 requested a full tablet report, sending...
I20260923 22:17:28.548513 8508 ts_manager.cc:194] Registered new tserver with Master: c2fbb7fb4c134f6787589e521d0790cb (127.4.207.3:40265)
I20260923 22:17:28.548866 8508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:38577
I20260923 22:17:28.551492 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:28.575579 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.566893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.4.207.1:46019)
0923 22:17:28.567074 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.567087 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.567099 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:28.567132 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.567134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.567149 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.567228 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.567885 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.568649 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.569502 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.570082 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.570183 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.572577 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.572585 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.572586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.572593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.574255 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.574663 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.574665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.574666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.574723 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.574936 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.574937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.574937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.575084 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.575191 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.575300 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.575364 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:28.584244 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:28.576900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.4.207.62:43043)
0923 22:17:28.576941 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:28.576943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:28.576954 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:28.577019 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:28.577020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:28.577030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:28.577069 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:28.577487 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.577991 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.578685 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:28.578875 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:28.578938 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:28.581753 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:28.581755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:28.581755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:28.581762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:28.583042 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.583416 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.583417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.583417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.583539 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:28.583735 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:28.583735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:28.583736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:28.583886 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:28.583963 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:28.584089 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 22:17:28.584175 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 22:17:28.584529 8508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56440
I20260923 22:17:28.584967 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 8570
I20260923 22:17:28.590849 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 8704
I20260923 22:17:28.595827 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 8838
I20260923 22:17:28.600754 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 8476
2026-09-23T22:17:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1927 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8979](info): set up 2 sockets
Sep 23 22:17:28 dist-test-slave-7511 krb5kdc[8979](info): commencing operation
krb5kdc: starting...
W20260923 22:17:29.627481 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:29 dist-test-slave-7511 krb5kdc[8979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:29.750965 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:45227
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:45227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:29.832877 8992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:29.833096 8992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:29.833118 8992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:29.834159 8992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:29.834192 8992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:29.834205 8992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:29.834218 8992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:29.834228 8992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:29.835692 8992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:45227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:45227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:29.835906 8992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:29.836093 8992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:29.840950 9000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.841017 8997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:29.840979 8998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:29.841145 8992 server_base.cc:1034] running on GCE node
I20260923 22:17:29.841359 8992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:29.841590 8992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:29.842823 8992 hybrid_clock.cc:648] HybridClock initialized: now 1790201849842808 us; error 28 us; skew 500 ppm
Sep 23 22:17:29 dist-test-slave-7511 krb5kdc[8979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:29.845448 8992 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:29.845813 8992 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:29.845863 8992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:29.845952 8992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:29.846989 8992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "fea81e7105584c34bb4e4e1669f81c5e"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-7511"
server_key: "ad5f174cfe5a5c7b8db943906c88f9af"
server_key_iv: "7d979d39e64c65311738b7da74b59391"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.847296 8992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "fea81e7105584c34bb4e4e1669f81c5e"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-7511"
server_key: "ad5f174cfe5a5c7b8db943906c88f9af"
server_key_iv: "7d979d39e64c65311738b7da74b59391"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.848608 8992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:29.849315 9006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:29.849536 8992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:29.849612 8992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "fea81e7105584c34bb4e4e1669f81c5e"
format_stamp: "Formatted at 2026-09-23 22:17:29 on dist-test-slave-7511"
server_key: "ad5f174cfe5a5c7b8db943906c88f9af"
server_key_iv: "7d979d39e64c65311738b7da74b59391"
server_key_version: "encryptionkey@0"
I20260923 22:17:29.849704 8992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:29.866443 8992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:29.868933 8992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:29.869110 8992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:29.873122 8992 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:45227
I20260923 22:17:29.873152 9058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:45227 every 8 connection(s)
I20260923 22:17:29.873448 8992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:29.874362 9059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:29.875871 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 8992
I20260923 22:17:29.875993 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:29.876181 4924 external_mini_cluster.cc:1428] Setting key 87753d66d4707651a79369ba46a2d385
I20260923 22:17:29.877254 9059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e: Bootstrap starting.
I20260923 22:17:29.877964 9059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:29.878311 9059 log.cc:824] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:29.879022 9059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e: No bootstrap required, opened a new log
I20260923 22:17:29.881259 9059 raft_consensus.cc:348] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea81e7105584c34bb4e4e1669f81c5e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45227 } }
I20260923 22:17:29.881429 9059 raft_consensus.cc:374] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:29.881454 9059 raft_consensus.cc:729] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea81e7105584c34bb4e4e1669f81c5e, State: Initialized, Role: FOLLOWER
I20260923 22:17:29.881608 9059 consensus_queue.cc:260] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea81e7105584c34bb4e4e1669f81c5e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45227 } }
I20260923 22:17:29.881702 9059 raft_consensus.cc:388] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:29.881748 9059 raft_consensus.cc:482] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:29.881798 9059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:29.882301 9059 raft_consensus.cc:504] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea81e7105584c34bb4e4e1669f81c5e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45227 } }
I20260923 22:17:29.882421 9059 leader_election.cc:302] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fea81e7105584c34bb4e4e1669f81c5e; no voters:
I20260923 22:17:29.882616 9059 leader_election.cc:288] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:29.882637 9064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:29.882748 9064 raft_consensus.cc:686] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [term 1 LEADER]: Becoming Leader. State: Replica: fea81e7105584c34bb4e4e1669f81c5e, State: Running, Role: LEADER
I20260923 22:17:29.882827 9064 consensus_queue.cc:237] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea81e7105584c34bb4e4e1669f81c5e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45227 } }
I20260923 22:17:29.882884 9059 sys_catalog.cc:564] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:29.883131 9064 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fea81e7105584c34bb4e4e1669f81c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea81e7105584c34bb4e4e1669f81c5e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45227 } } }
I20260923 22:17:29.883200 9064 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [sys.catalog]: This master's current role is: LEADER
Sep 23 22:17:29 dist-test-slave-7511 krb5kdc[8979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:29.883329 9064 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fea81e7105584c34bb4e4e1669f81c5e. Latest consensus state: current_term: 1 leader_uuid: "fea81e7105584c34bb4e4e1669f81c5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fea81e7105584c34bb4e4e1669f81c5e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45227 } } }
I20260923 22:17:29.883400 9064 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:29.884409 9068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:29.885169 9068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:29.886684 9068 catalog_manager.cc:1347] Generated new cluster ID: 36954e0d2196493d8bbb15148cbb5755
I20260923 22:17:29.886755 9068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:29.887697 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:29.876734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.4.207.62:45227)
0923 22:17:29.876940 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:29.876954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:29.876980 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:17:29.877441 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:29.877448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:29.877498 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:29.877622 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:29.878458 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.879548 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.880604 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:29.880988 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:29.881068 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:29.883796 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:29.883800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:29.883800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:29.883814 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:29.885906 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.886322 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.886325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.886326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.886426 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:29.886677 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:29.886678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:29.886678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:29.886998 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:29.887184 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:29.887293 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 22:17:29.887550 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260923 22:17:29.896016 9068 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:29.896757 9068 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:29.905675 9068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e: Generated new TSK 0
I20260923 22:17:29.905884 9068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:29.936733 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45227
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:30.019079 9085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.019313 9085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.019354 9085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.020540 9085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.020609 9085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.020653 9085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:30.022267 9085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:30.022568 9085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.022821 9085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.027864 9093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.027899 9091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.027880 9090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.028172 9085 server_base.cc:1034] running on GCE node
I20260923 22:17:30.028386 9085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.028654 9085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.029834 9085 hybrid_clock.cc:648] HybridClock initialized: now 1790201850029822 us; error 27 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-7511 krb5kdc[8979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.032536 9085 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:30.032941 9085 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.033016 9085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.033174 9085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:30.034260 9085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "704caa18a5234274b705fdb38620f4c7"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "b675cc55df64c92519e9bbb1d61b8b5c"
server_key_iv: "6b09126bee1db6e50887c3d85e9c3b50"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.034639 9085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "704caa18a5234274b705fdb38620f4c7"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "b675cc55df64c92519e9bbb1d61b8b5c"
server_key_iv: "6b09126bee1db6e50887c3d85e9c3b50"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.036037 9085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:30.036847 9099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.037017 9085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.037155 9085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "704caa18a5234274b705fdb38620f4c7"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "b675cc55df64c92519e9bbb1d61b8b5c"
server_key_iv: "6b09126bee1db6e50887c3d85e9c3b50"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.037237 9085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.061064 9085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.064334 9085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.064528 9085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.064838 9085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:30.065209 9085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:30.065246 9085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.065276 9085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:30.065325 9085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.071642 9085 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:34949
I20260923 22:17:30.071668 9212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:34949 every 8 connection(s)
I20260923 22:17:30.071970 9085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:30.072235 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 9085
I20260923 22:17:30.072355 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:30.072552 4924 external_mini_cluster.cc:1428] Setting key 9c5fe67ff54ee30f33c3919bfc31a176
Sep 23 22:17:30 dist-test-slave-7511 krb5kdc[8979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
I20260923 22:17:30.083973 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.072738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:41855 (local address 127.4.207.62:45227)
0923 22:17:30.072794 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.072796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.073034 (+ 238us) server_negotiation.cc:407] Connection header received
0923 22:17:30.073605 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.073606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.073615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.073662 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.075198 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.076065 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.077725 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.077959 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.078019 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.080644 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.080646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.080646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.080655 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.082293 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.082724 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.082725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.082726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.082805 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.083131 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.083132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.083132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.083257 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.083364 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.083683 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.083767 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:30.084124 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.072713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:45227 (local address 127.4.207.1:41855)
0923 22:17:30.072901 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.072943 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.073499 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.073750 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.073754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.074227 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.075062 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.075070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.076181 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.076182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.077618 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.077622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.078056 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.078057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.078233 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.078812 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.078822 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.080529 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.082386 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.082395 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.082397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.082641 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.082874 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.082875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.082876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.083027 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.083359 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.083365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.083558 (+ 193us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.083657 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.083863 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
I20260923 22:17:30.084668 9213 heartbeater.cc:347] Connected to a master server at 127.4.207.62:45227
I20260923 22:17:30.084774 9213 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.084951 9213 heartbeater.cc:511] Master 127.4.207.62:45227 requested a full tablet report, sending...
I20260923 22:17:30.085608 9023 ts_manager.cc:194] Registered new tserver with Master: 704caa18a5234274b705fdb38620f4c7 (127.4.207.1:34949)
I20260923 22:17:30.086565 9023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:41855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:30.100270 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45227
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:30.183715 9218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.183934 9218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.183985 9218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.185236 9218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.185288 9218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.185318 9218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:30.186810 9218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:30.187005 9218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.187223 9218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.192395 9226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.192395 9223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.192462 9224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.192746 9218 server_base.cc:1034] running on GCE node
I20260923 22:17:30.192973 9218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.193228 9218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.194371 9218 hybrid_clock.cc:648] HybridClock initialized: now 1790201850194349 us; error 37 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-7511 krb5kdc[8979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.196969 9218 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:30.197367 9218 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.197418 9218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.197584 9218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:30.198674 9218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "74d48972e258457395350b24e4779b5d"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "b28372e5322734453178c2e408b837ac"
server_key_iv: "07b0c0de5336da3c53fcccb1c009813a"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.199059 9218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "74d48972e258457395350b24e4779b5d"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "b28372e5322734453178c2e408b837ac"
server_key_iv: "07b0c0de5336da3c53fcccb1c009813a"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.200485 9218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:30.201220 9232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.201393 9218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:30.201500 9218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "74d48972e258457395350b24e4779b5d"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "b28372e5322734453178c2e408b837ac"
server_key_iv: "07b0c0de5336da3c53fcccb1c009813a"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.201576 9218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.225215 9218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.227222 9218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.227401 9218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.227671 9218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:30.228040 9218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:30.228129 9218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.228191 9218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:30.228219 9218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.234236 9218 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:43943
I20260923 22:17:30.234274 9345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:43943 every 8 connection(s)
I20260923 22:17:30.234558 9218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:30.235167 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 9218
I20260923 22:17:30.235302 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:30.235558 4924 external_mini_cluster.cc:1428] Setting key 98a958cf180d1e6f1b52e8ce22921d86
Sep 23 22:17:30 dist-test-slave-7511 krb5kdc[8979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
I20260923 22:17:30.245000 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.235322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:40541 (local address 127.4.207.62:45227)
0923 22:17:30.235371 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.235373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.235631 (+ 258us) server_negotiation.cc:407] Connection header received
0923 22:17:30.236185 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.236186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.236197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.236247 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.237260 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.237881 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.238914 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.239115 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.239202 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.241839 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.241841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.241841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.241850 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.243485 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.243938 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.243939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.243940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.244003 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.244308 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.244309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.244309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.244436 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.244510 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.244810 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.244901 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:30.245318 9348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.235290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:45227 (local address 127.4.207.2:40541)
0923 22:17:30.235469 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.235513 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.236110 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.236333 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.236338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.236628 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.237150 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.237155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.237980 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.237981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.238802 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.238805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.239213 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.239214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.239341 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.239945 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.239989 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.241732 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.243581 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.243588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.243591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.243819 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.244073 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.244074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.244075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.244232 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.244514 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.244521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.244675 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.244788 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.244970 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
I20260923 22:17:30.245832 9346 heartbeater.cc:347] Connected to a master server at 127.4.207.62:45227
I20260923 22:17:30.245932 9346 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.246055 9346 heartbeater.cc:511] Master 127.4.207.62:45227 requested a full tablet report, sending...
I20260923 22:17:30.246446 9023 ts_manager.cc:194] Registered new tserver with Master: 74d48972e258457395350b24e4779b5d (127.4.207.2:43943)
I20260923 22:17:30.246853 9023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:40541
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:30.262372 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45227
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:30.344898 9351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.345139 9351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.345166 9351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.346383 9351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.346429 9351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.346457 9351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:30.348134 9351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:45227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:30.348404 9351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.348654 9351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.353983 9359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.354007 9357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.354074 9351 server_base.cc:1034] running on GCE node
W20260923 22:17:30.353984 9356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.354370 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.354635 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.355801 9351 hybrid_clock.cc:648] HybridClock initialized: now 1790201850355788 us; error 43 us; skew 500 ppm
Sep 23 22:17:30 dist-test-slave-7511 krb5kdc[8979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:30.358469 9351 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:30.358829 9351 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.358901 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.359054 9351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:30.360206 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "9b55c7c82b71458ea85ceeabad0cb72a"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "552346b969c55efd3050fff8cccae312"
server_key_iv: "e39c0a4fd6c50eda0289fca69f5828ec"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.360574 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "9b55c7c82b71458ea85ceeabad0cb72a"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "552346b969c55efd3050fff8cccae312"
server_key_iv: "e39c0a4fd6c50eda0289fca69f5828ec"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.361874 9351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:30.362643 9365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.362834 9351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.362941 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "9b55c7c82b71458ea85ceeabad0cb72a"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "552346b969c55efd3050fff8cccae312"
server_key_iv: "e39c0a4fd6c50eda0289fca69f5828ec"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.363015 9351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.377285 9351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.379741 9351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.379900 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.380131 9351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:30.380450 9351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:30.380502 9351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.380546 9351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:30.380575 9351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.386821 9351 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:39045
I20260923 22:17:30.386888 9478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:39045 every 8 connection(s)
I20260923 22:17:30.387179 9351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:30.387583 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 9351
I20260923 22:17:30.387694 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:30.387912 4924 external_mini_cluster.cc:1428] Setting key 7f096c9343ef74d71a7ad5d2e6e0c938
Sep 23 22:17:30 dist-test-slave-7511 krb5kdc[8979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201850, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:30.396546 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.388063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:39795 (local address 127.4.207.62:45227)
0923 22:17:30.388112 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.388114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.388497 (+ 383us) server_negotiation.cc:407] Connection header received
0923 22:17:30.389175 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.389176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.389189 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.389240 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.390202 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.390701 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.391764 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.391973 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.392058 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.394120 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.394121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.394122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.394130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.395322 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.395772 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.395773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.395773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.395823 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.396026 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.396027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.396027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.396128 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.396175 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.396418 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.396481 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:30.396761 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.388071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:45227 (local address 127.4.207.3:39795)
0923 22:17:30.388358 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.388402 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.389059 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.389311 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.389315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.389589 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.390100 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.390105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.390804 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.390805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.391680 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.391683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.392066 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.392067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.392194 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.392648 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:30.392655 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.394034 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.395395 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.395401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.395403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.395710 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.395881 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:30.395882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:30.395882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:30.395975 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:30.396195 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.396199 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.396313 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.396395 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.396511 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":89,"threads_started":1}
I20260923 22:17:30.397289 9479 heartbeater.cc:347] Connected to a master server at 127.4.207.62:45227
I20260923 22:17:30.397370 9479 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.397531 9479 heartbeater.cc:511] Master 127.4.207.62:45227 requested a full tablet report, sending...
I20260923 22:17:30.397940 9023 ts_manager.cc:194] Registered new tserver with Master: 9b55c7c82b71458ea85ceeabad0cb72a (127.4.207.3:39045)
I20260923 22:17:30.398286 9023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:39795
I20260923 22:17:30.399690 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:30.407230 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.400845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.4.207.62:45227)
0923 22:17:30.400911 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.400913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.400979 (+ 66us) server_negotiation.cc:407] Connection header received
0923 22:17:30.401051 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.401052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.401061 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.401129 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.401654 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.402135 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.402848 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.403038 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.403121 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.404295 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.404296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.404296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:30.404303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.405935 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.406325 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.406326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.406326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.406381 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:30.406602 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:30.406603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:30.406604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:30.406780 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.406887 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.407033 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.407127 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:17:30.408787 9023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:30.410430 9023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:30.417135 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.412953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:43943 (local address 127.0.0.1:57654)
0923 22:17:30.413206 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.413216 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.413259 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.413868 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.413868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.413871 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.413916 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.413918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.414768 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.414769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.416151 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.416152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.416830 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.416907 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.416920 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.416986 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.417043 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":80,"thread_start_us":202,"threads_started":1}
I20260923 22:17:30.417248 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.412744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:39045 (local address 127.0.0.1:37410)
0923 22:17:30.412858 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.412888 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.412945 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.413312 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.413315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.413317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.413386 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.413388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.414851 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.414851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.416149 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.416152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.416962 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.417019 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.417022 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.417085 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.417120 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":39,"thread_start_us":76,"threads_started":1}
I20260923 22:17:30.417322 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.413502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.4.207.1:34949)
0923 22:17:30.413643 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.413648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.413657 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.413695 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.413697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.413710 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.413853 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.413987 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.414973 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416088 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.416796 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416868 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.416928 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.417188 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.417224 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260923 22:17:30.417459 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.413254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:34949 (local address 127.0.0.1:50636)
0923 22:17:30.413376 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.413387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.413435 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.413868 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.413868 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.413872 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.413916 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.413919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.415068 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.415069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.416004 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.416006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.416960 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.417024 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.417028 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.417075 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.417116 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
I20260923 22:17:30.417526 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.413141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.4.207.2:43943)
0923 22:17:30.413527 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.413532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.413541 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.413591 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.413593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.413686 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.413761 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.413999 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.414668 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416247 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.416728 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416826 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.416890 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.417369 (+ 479us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.417436 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":75,"threads_started":1}
I20260923 22:17:30.418308 9147 tablet_service.cc:1467] Processing CreateTablet for tablet e21bd5f234324775a9065914d443ec87 (DEFAULT_TABLE table=test-table [id=f91ec94c308640ccba1e5d00af2f55b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:30.418308 9280 tablet_service.cc:1467] Processing CreateTablet for tablet e21bd5f234324775a9065914d443ec87 (DEFAULT_TABLE table=test-table [id=f91ec94c308640ccba1e5d00af2f55b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:30.418679 9147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21bd5f234324775a9065914d443ec87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.418679 9280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21bd5f234324775a9065914d443ec87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.422067 9497 tablet_bootstrap.cc:492] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Bootstrap starting.
I20260923 22:17:30.422152 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.412858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.4.207.3:39045)
0923 22:17:30.413040 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.413046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.413055 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:30.413092 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.413094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.413108 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.413206 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.413652 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.414768 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416229 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.416864 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.416955 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.417002 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.417100 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.417142 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260923 22:17:30.422138 9496 tablet_bootstrap.cc:492] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7: Bootstrap starting.
I20260923 22:17:30.422819 9496 tablet_bootstrap.cc:654] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.422857 9497 tablet_bootstrap.cc:654] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.422766 9413 tablet_service.cc:1467] Processing CreateTablet for tablet e21bd5f234324775a9065914d443ec87 (DEFAULT_TABLE table=test-table [id=f91ec94c308640ccba1e5d00af2f55b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:30.423099 9413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e21bd5f234324775a9065914d443ec87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.423574 9497 log.cc:824] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.423576 9496 log.cc:824] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.424476 9497 tablet_bootstrap.cc:492] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: No bootstrap required, opened a new log
I20260923 22:17:30.424543 9497 ts_tablet_manager.cc:1397] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 22:17:30.424660 9496 tablet_bootstrap.cc:492] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7: No bootstrap required, opened a new log
I20260923 22:17:30.424717 9496 ts_tablet_manager.cc:1397] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 22:17:30.426246 9500 tablet_bootstrap.cc:492] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: Bootstrap starting.
I20260923 22:17:30.426641 9497 raft_consensus.cc:348] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.426641 9496 raft_consensus.cc:348] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.426826 9496 raft_consensus.cc:374] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.426826 9497 raft_consensus.cc:374] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.426863 9497 raft_consensus.cc:729] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d48972e258457395350b24e4779b5d, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.426863 9496 raft_consensus.cc:729] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704caa18a5234274b705fdb38620f4c7, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.426867 9500 tablet_bootstrap.cc:654] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.426990 9497 consensus_queue.cc:260] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.427034 9496 consensus_queue.cc:260] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.427300 9497 ts_tablet_manager.cc:1428] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:17:30.427300 9496 ts_tablet_manager.cc:1428] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:17:30.427420 9213 heartbeater.cc:503] Master 127.4.207.62:45227 was elected leader, sending a full tablet report...
I20260923 22:17:30.427484 9346 heartbeater.cc:503] Master 127.4.207.62:45227 was elected leader, sending a full tablet report...
I20260923 22:17:30.427515 9500 log.cc:824] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.428297 9500 tablet_bootstrap.cc:492] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: No bootstrap required, opened a new log
I20260923 22:17:30.428375 9500 ts_tablet_manager.cc:1397] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:17:30.429883 9500 raft_consensus.cc:348] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.430014 9500 raft_consensus.cc:374] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.430035 9500 raft_consensus.cc:729] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b55c7c82b71458ea85ceeabad0cb72a, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.430154 9500 consensus_queue.cc:260] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.430411 9500 ts_tablet_manager.cc:1428] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:17:30.430444 9479 heartbeater.cc:503] Master 127.4.207.62:45227 was elected leader, sending a full tablet report...
I20260923 22:17:30.481567 9502 raft_consensus.cc:482] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:30.481699 9502 raft_consensus.cc:504] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.482170 9502 leader_election.cc:288] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b55c7c82b71458ea85ceeabad0cb72a (127.4.207.3:39045), 74d48972e258457395350b24e4779b5d (127.4.207.2:43943)
W20260923 22:17:30.485037 9347 tablet.cc:2369] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:30.486210 9215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.482357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:39045 (local address 127.4.207.1:36577)
0923 22:17:30.482418 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.482431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.482479 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.482731 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.482731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.482734 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.482823 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.482825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.483835 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.483835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.485141 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.485144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.485930 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.485931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.486020 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.486023 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.486087 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.486131 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":48}
I20260923 22:17:30.486239 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.482466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:36577 (local address 127.4.207.3:39045)
0923 22:17:30.482535 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.482537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.482547 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:30.482594 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.482595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.482604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.482718 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.482922 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.483754 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.485255 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.485820 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.485884 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.485893 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.486120 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.486175 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:17:30.486274 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.482601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:42201 (local address 127.4.207.2:43943)
0923 22:17:30.482669 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.482671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.482747 (+ 76us) server_negotiation.cc:407] Connection header received
0923 22:17:30.482820 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.482821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.482833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.482891 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:30.483067 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.483804 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.485265 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.485820 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.485867 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.485878 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.486170 (+ 292us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.486216 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:17:30.486454 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.482490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:43943 (local address 127.4.207.1:42201)
0923 22:17:30.482673 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.482685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.482739 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.482919 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.482920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.482921 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:30.482983 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.482985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.483882 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.483882 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.485141 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:30.485144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.485926 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.485927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:30.486033 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.486036 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.486340 (+ 304us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.486386 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":8,"thread_start_us":119,"threads_started":1}
I20260923 22:17:30.486847 9300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21bd5f234324775a9065914d443ec87" candidate_uuid: "704caa18a5234274b705fdb38620f4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d48972e258457395350b24e4779b5d" is_pre_election: true
I20260923 22:17:30.486851 9433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21bd5f234324775a9065914d443ec87" candidate_uuid: "704caa18a5234274b705fdb38620f4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" is_pre_election: true
I20260923 22:17:30.486972 9300 raft_consensus.cc:2454] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 704caa18a5234274b705fdb38620f4c7 in term 0.
I20260923 22:17:30.487012 9433 raft_consensus.cc:2454] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 704caa18a5234274b705fdb38620f4c7 in term 0.
I20260923 22:17:30.487242 9100 leader_election.cc:302] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 704caa18a5234274b705fdb38620f4c7, 9b55c7c82b71458ea85ceeabad0cb72a; no voters:
I20260923 22:17:30.487407 9502 raft_consensus.cc:2781] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:30.487497 9502 raft_consensus.cc:482] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:30.487529 9502 raft_consensus.cc:3037] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.488276 9502 raft_consensus.cc:504] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.488384 9502 leader_election.cc:288] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 9b55c7c82b71458ea85ceeabad0cb72a (127.4.207.3:39045), 74d48972e258457395350b24e4779b5d (127.4.207.2:43943)
I20260923 22:17:30.488577 9433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21bd5f234324775a9065914d443ec87" candidate_uuid: "704caa18a5234274b705fdb38620f4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b55c7c82b71458ea85ceeabad0cb72a"
I20260923 22:17:30.488642 9300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e21bd5f234324775a9065914d443ec87" candidate_uuid: "704caa18a5234274b705fdb38620f4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74d48972e258457395350b24e4779b5d"
I20260923 22:17:30.488692 9433 raft_consensus.cc:3037] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.488708 9300 raft_consensus.cc:3037] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.489669 9300 raft_consensus.cc:2454] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704caa18a5234274b705fdb38620f4c7 in term 1.
I20260923 22:17:30.489669 9433 raft_consensus.cc:2454] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 704caa18a5234274b705fdb38620f4c7 in term 1.
I20260923 22:17:30.489846 9100 leader_election.cc:302] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 704caa18a5234274b705fdb38620f4c7, 74d48972e258457395350b24e4779b5d; no voters:
I20260923 22:17:30.489960 9502 raft_consensus.cc:2781] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:30.490145 9502 raft_consensus.cc:686] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 1 LEADER]: Becoming Leader. State: Replica: 704caa18a5234274b705fdb38620f4c7, State: Running, Role: LEADER
I20260923 22:17:30.490254 9502 consensus_queue.cc:237] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } }
I20260923 22:17:30.491173 9023 catalog_manager.cc:5495] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 704caa18a5234274b705fdb38620f4c7 (127.4.207.1). New cstate: current_term: 1 leader_uuid: "704caa18a5234274b705fdb38620f4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "704caa18a5234274b705fdb38620f4c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 34949 } health_report { overall_health: HEALTHY } } }
I20260923 22:17:30.509207 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.506648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.4.207.1:34949)
0923 22:17:30.506720 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.506723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.506752 (+ 29us) server_negotiation.cc:407] Connection header received
0923 22:17:30.506790 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.506791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.506813 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.506896 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:30.507049 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.507599 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.508390 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:30.508598 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:30.508678 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:30.508848 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:30.508965 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:30.509059 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.509078 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.509098 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:17:30.511066 9300 raft_consensus.cc:1261] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 1 FOLLOWER]: Refusing update from remote peer 704caa18a5234274b705fdb38620f4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:30.511098 9433 raft_consensus.cc:1261] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 1 FOLLOWER]: Refusing update from remote peer 704caa18a5234274b705fdb38620f4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:30.511524 9502 consensus_queue.cc:1035] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74d48972e258457395350b24e4779b5d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:30.511667 9506 consensus_queue.cc:1035] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b55c7c82b71458ea85ceeabad0cb72a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:30.514513 9510 mvcc.cc:205] Tried to move back new op lower bound from 7332666779690504192 to 7332666779608801280. Current Snapshot: MvccSnapshot[applied={T|T < 7332666779690504192}]
I20260923 22:17:30.514636 9511 mvcc.cc:205] Tried to move back new op lower bound from 7332666779690504192 to 7332666779608801280. Current Snapshot: MvccSnapshot[applied={T|T < 7332666779690504192}]
I20260923 22:17:30.516376 9023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:30.516469 9023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:30.517233 9023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fea81e7105584c34bb4e4e1669f81c5e: Sending DeleteTablet for 3 replicas of tablet e21bd5f234324775a9065914d443ec87
I20260923 22:17:30.517535 9413 tablet_service.cc:1514] Processing DeleteTablet for tablet e21bd5f234324775a9065914d443ec87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:30 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:37410
I20260923 22:17:30.517586 9147 tablet_service.cc:1514] Processing DeleteTablet for tablet e21bd5f234324775a9065914d443ec87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:30 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:50636
I20260923 22:17:30.517599 9280 tablet_service.cc:1514] Processing DeleteTablet for tablet e21bd5f234324775a9065914d443ec87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:30 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:57654
I20260923 22:17:30.517788 9520 tablet_replica.cc:330] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7: stopping tablet replica
I20260923 22:17:30.517887 9520 raft_consensus.cc:2229] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:30.518008 9520 raft_consensus.cc:2258] T e21bd5f234324775a9065914d443ec87 P 704caa18a5234274b705fdb38620f4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:30.518097 9519 tablet_replica.cc:330] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: stopping tablet replica
I20260923 22:17:30.518177 9519 raft_consensus.cc:2229] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:30.518229 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 9085
I20260923 22:17:30.518262 9519 raft_consensus.cc:2258] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:30.519131 9521 tablet_replica.cc:330] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: stopping tablet replica
I20260923 22:17:30.519222 9521 raft_consensus.cc:2229] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:30.519330 9521 raft_consensus.cc:2258] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:30.519929 9521 ts_tablet_manager.cc:1905] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:30.522050 9521 ts_tablet_manager.cc:1918] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:30.522132 9521 log.cc:1196] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/e21bd5f234324775a9065914d443ec87
I20260923 22:17:30.522413 9521 ts_tablet_manager.cc:1939] T e21bd5f234324775a9065914d443ec87 P 74d48972e258457395350b24e4779b5d: Deleting consensus metadata
I20260923 22:17:30.523098 9007 catalog_manager.cc:4841] TS 74d48972e258457395350b24e4779b5d (127.4.207.2:43943): tablet e21bd5f234324775a9065914d443ec87 (table test-table [id=f91ec94c308640ccba1e5d00af2f55b2]) successfully deleted
I20260923 22:17:30.524497 9519 ts_tablet_manager.cc:1905] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 22:17:30.524770 9007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:34949 (error 108)
I20260923 22:17:30.524957 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 9218
I20260923 22:17:30.525593 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.525170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:34949 (local address 127.0.0.1:50652)
0923 22:17:30.525517 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.525555 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:34949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
W20260923 22:17:30.525728 9007 catalog_manager.cc:4584] TS 704caa18a5234274b705fdb38620f4c7 (127.4.207.1:34949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e21bd5f234324775a9065914d443ec87: Network error: Client connection negotiation failed: client connection to 127.4.207.1:34949: connect: Connection refused (error 111)
I20260923 22:17:30.526885 9519 ts_tablet_manager.cc:1918] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:30.526939 9519 log.cc:1196] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/e21bd5f234324775a9065914d443ec87
I20260923 22:17:30.527189 9519 ts_tablet_manager.cc:1939] T e21bd5f234324775a9065914d443ec87 P 9b55c7c82b71458ea85ceeabad0cb72a: Deleting consensus metadata
I20260923 22:17:30.527608 9007 catalog_manager.cc:4841] TS 9b55c7c82b71458ea85ceeabad0cb72a (127.4.207.3:39045): tablet e21bd5f234324775a9065914d443ec87 (table test-table [id=f91ec94c308640ccba1e5d00af2f55b2]) successfully deleted
I20260923 22:17:30.530025 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 9351
I20260923 22:17:30.534976 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 8992
2026-09-23T22:17:30Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1932 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T22:17:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:30Z Disabled control of system clock
I20260923 22:17:30.562557 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:37367
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 22:17:30.642383 9526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.642596 9526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.642616 9526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.643721 9526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:30.643755 9526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.643767 9526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.643780 9526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:30.643792 9526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:30.645079 9526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:37367
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:30.645287 9526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.645524 9526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.650588 9531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.650569 9532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.650599 9526 server_base.cc:1034] running on GCE node
W20260923 22:17:30.650569 9534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.651087 9526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.651473 9526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.652634 9526 hybrid_clock.cc:648] HybridClock initialized: now 1790201850652609 us; error 38 us; skew 500 ppm
I20260923 22:17:30.654412 9526 webserver.cc:460] Webserver started at http://127.4.207.62:45789/ using document root <none> and password file <none>
I20260923 22:17:30.654695 9526 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.654750 9526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.654868 9526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:30.656059 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "2954a03a6e9a430d8f3cad91f2d3499c"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "28b3650a4df1c602a697a384cd4f07a4"
server_key_iv: "fd97fc4e190db31cd625368f3976cc06"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.656454 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "2954a03a6e9a430d8f3cad91f2d3499c"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "28b3650a4df1c602a697a384cd4f07a4"
server_key_iv: "fd97fc4e190db31cd625368f3976cc06"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.657769 9526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:30.658533 9540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.658725 9526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.658833 9526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "2954a03a6e9a430d8f3cad91f2d3499c"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "28b3650a4df1c602a697a384cd4f07a4"
server_key_iv: "fd97fc4e190db31cd625368f3976cc06"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.658926 9526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.662501 9526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.662813 9526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.662968 9526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.667300 9526 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:37367
I20260923 22:17:30.667330 9592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:37367 every 8 connection(s)
I20260923 22:17:30.667758 9526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:30.668465 9593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:30.671341 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c: Bootstrap starting.
I20260923 22:17:30.671998 9593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:30.672305 9593 log.cc:824] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:30.673087 9593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c: No bootstrap required, opened a new log
I20260923 22:17:30.674710 9593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2954a03a6e9a430d8f3cad91f2d3499c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 37367 } }
I20260923 22:17:30.674854 9593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:30.674878 9593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2954a03a6e9a430d8f3cad91f2d3499c, State: Initialized, Role: FOLLOWER
I20260923 22:17:30.675045 9593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2954a03a6e9a430d8f3cad91f2d3499c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 37367 } }
I20260923 22:17:30.675112 9593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:30.675134 9593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:30.675204 9593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:30.675853 9593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2954a03a6e9a430d8f3cad91f2d3499c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 37367 } }
I20260923 22:17:30.675951 9593 leader_election.cc:302] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2954a03a6e9a430d8f3cad91f2d3499c; no voters:
I20260923 22:17:30.676182 9593 leader_election.cc:288] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:30.676326 9596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:30.676518 9596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [term 1 LEADER]: Becoming Leader. State: Replica: 2954a03a6e9a430d8f3cad91f2d3499c, State: Running, Role: LEADER
I20260923 22:17:30.676537 9593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:30.676772 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 9526
I20260923 22:17:30.676884 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:30.677062 4924 external_mini_cluster.cc:1428] Setting key 02994f2067dbec288cbd89aee7652d8e
I20260923 22:17:30.677696 9596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2954a03a6e9a430d8f3cad91f2d3499c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 37367 } }
I20260923 22:17:30.678086 9598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2954a03a6e9a430d8f3cad91f2d3499c. Latest consensus state: current_term: 1 leader_uuid: "2954a03a6e9a430d8f3cad91f2d3499c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2954a03a6e9a430d8f3cad91f2d3499c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 37367 } } }
I20260923 22:17:30.678184 9598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:30.678273 9597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2954a03a6e9a430d8f3cad91f2d3499c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2954a03a6e9a430d8f3cad91f2d3499c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 37367 } } }
I20260923 22:17:30.678342 9597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:30.679188 9612 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:30.679656 9612 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:30.679741 9613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:30.679899 9613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:30.680305 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.677627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.4.207.62:37367)
0923 22:17:30.677992 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.678002 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.678036 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:17:30.678783 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.678792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.678846 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.679011 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.679817 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.679820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.679821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:30.679826 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.679869 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:30.679883 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.679963 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.680055 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.680129 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":81,"threads_started":1}
I20260923 22:17:30.681717 9613 catalog_manager.cc:1347] Generated new cluster ID: e1ce356980444171b4d79e1182ead051
I20260923 22:17:30.681769 9613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:30.691341 9613 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:30.691578 9613 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:30.696123 9613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c: Generated new TSK 0
I20260923 22:17:30.696331 9613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:30.699810 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:37367
--builtin_ntp_servers=127.4.207.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 22:17:30.785061 9617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.785290 9617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.785316 9617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.786571 9617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.786643 9617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.786674 9617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:30.788187 9617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:30.788457 9617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.788805 9617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.794818 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.794936 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.794819 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.795246 9617 server_base.cc:1034] running on GCE node
I20260923 22:17:30.795481 9617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.795766 9617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.796939 9617 hybrid_clock.cc:648] HybridClock initialized: now 1790201850796928 us; error 33 us; skew 500 ppm
I20260923 22:17:30.798352 9617 webserver.cc:460] Webserver started at http://127.4.207.1:34415/ using document root <none> and password file <none>
I20260923 22:17:30.798607 9617 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.798678 9617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.798834 9617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:30.800000 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "2d78fab6619642caaaa33ed3f4574a86"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "bbdd8a2a09c0d48e04cbf3f3f7f37764"
server_key_iv: "8523201cc14f502fa61702cad80167e2"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.800367 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "2d78fab6619642caaaa33ed3f4574a86"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "bbdd8a2a09c0d48e04cbf3f3f7f37764"
server_key_iv: "8523201cc14f502fa61702cad80167e2"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.801723 9617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:30.802479 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.802711 9617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.802830 9617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "2d78fab6619642caaaa33ed3f4574a86"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "bbdd8a2a09c0d48e04cbf3f3f7f37764"
server_key_iv: "8523201cc14f502fa61702cad80167e2"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.802898 9617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.807672 9617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.807919 9617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.808017 9617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.808177 9617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:30.808422 9617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:30.808449 9617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.808470 9617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:30.808481 9617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.814036 9617 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:39727
I20260923 22:17:30.814075 9744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:39727 every 8 connection(s)
I20260923 22:17:30.814431 9617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:30.814908 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 9617
I20260923 22:17:30.815021 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:30.815198 4924 external_mini_cluster.cc:1428] Setting key 91f7a00023eafea42ee1d9d9ddd95d4e
I20260923 22:17:30.816308 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:37367
--builtin_ntp_servers=127.4.207.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 22:17:30.818148 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.815284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:40721 (local address 127.4.207.62:37367)
0923 22:17:30.815334 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.815336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.815680 (+ 344us) server_negotiation.cc:407] Connection header received
0923 22:17:30.816346 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.816347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.816353 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.816431 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.817862 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.817864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.817864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:30.817869 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.817884 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:30.817889 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.817936 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.818037 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.818061 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:30.818352 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.815303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:37367 (local address 127.4.207.1:40721)
0923 22:17:30.815525 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.815578 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.816239 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.816527 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.816531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.816541 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.817695 (+ 1154us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:30.817696 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.817740 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:30.817780 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.817943 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.817947 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.817950 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.818048 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.818128 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260923 22:17:30.818988 9745 heartbeater.cc:347] Connected to a master server at 127.4.207.62:37367
I20260923 22:17:30.819090 9745 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.819244 9745 heartbeater.cc:511] Master 127.4.207.62:37367 requested a full tablet report, sending...
I20260923 22:17:30.819783 9557 ts_manager.cc:194] Registered new tserver with Master: 2d78fab6619642caaaa33ed3f4574a86 (127.4.207.1:39727)
W20260923 22:17:30.899029 9748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:30.899295 9748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:30.899330 9748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:30.900537 9748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:30.900606 9748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:30.900645 9748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:30.901993 9748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:30.902275 9748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:30.902576 9748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:30.907634 9753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:30.907630 9756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.907879 9748 server_base.cc:1034] running on GCE node
W20260923 22:17:30.908058 9754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:30.908324 9748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:30.908629 9748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:30.909817 9748 hybrid_clock.cc:648] HybridClock initialized: now 1790201850909806 us; error 41 us; skew 500 ppm
I20260923 22:17:30.911271 9748 webserver.cc:460] Webserver started at http://127.4.207.2:45987/ using document root <none> and password file <none>
I20260923 22:17:30.911615 9748 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:30.911692 9748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:30.911859 9748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:30.913065 9748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "8f37b59f8ca6482ab711fd81e43f9a4c"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "64abcea60a9871e195d7354563643a4f"
server_key_iv: "5ac4aad36b6396d2f5f496dce8d8e21c"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.913502 9748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "8f37b59f8ca6482ab711fd81e43f9a4c"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "64abcea60a9871e195d7354563643a4f"
server_key_iv: "5ac4aad36b6396d2f5f496dce8d8e21c"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.914961 9748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:30.915709 9762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.915874 9748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:30.915956 9748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "8f37b59f8ca6482ab711fd81e43f9a4c"
format_stamp: "Formatted at 2026-09-23 22:17:30 on dist-test-slave-7511"
server_key: "64abcea60a9871e195d7354563643a4f"
server_key_iv: "5ac4aad36b6396d2f5f496dce8d8e21c"
server_key_version: "encryptionkey@0"
I20260923 22:17:30.916009 9748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:30.919445 9748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:30.919829 9748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:30.919987 9748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:30.920220 9748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:30.920571 9748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:30.920605 9748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.920655 9748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:30.920680 9748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:30.926333 9748 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:40001
I20260923 22:17:30.926406 9875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:40001 every 8 connection(s)
I20260923 22:17:30.926733 9748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:30.929756 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.927533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:40885 (local address 127.4.207.62:37367)
0923 22:17:30.927597 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:30.927599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:30.927875 (+ 276us) server_negotiation.cc:407] Connection header received
0923 22:17:30.928368 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:30.928369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:30.928376 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:30.928432 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:30.929532 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:30.929533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:30.929533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:30.929538 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:30.929554 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:30.929557 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:30.929617 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:30.929688 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 22:17:30.929701 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:17:30.929934 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:30.927529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:37367 (local address 127.4.207.2:40885)
0923 22:17:30.927725 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:30.927758 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:30.928272 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:30.928484 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:30.928487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:30.928495 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:30.929380 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:30.929381 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:30.929409 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:30.929437 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:30.929626 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:30.929628 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:30.929631 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:30.929685 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 22:17:30.929734 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260923 22:17:30.930477 9876 heartbeater.cc:347] Connected to a master server at 127.4.207.62:37367
I20260923 22:17:30.930603 9876 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:30.930753 9876 heartbeater.cc:511] Master 127.4.207.62:37367 requested a full tablet report, sending...
I20260923 22:17:30.931121 9557 ts_manager.cc:194] Registered new tserver with Master: 8f37b59f8ca6482ab711fd81e43f9a4c (127.4.207.2:40001)
I20260923 22:17:30.931982 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 9748
I20260923 22:17:30.932111 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:30.932292 4924 external_mini_cluster.cc:1428] Setting key 4e81e48c20b25bcbbffd1f6f494e1065
I20260923 22:17:30.933193 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:37367
--builtin_ntp_servers=127.4.207.20:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 22:17:31.013898 9879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:31.014107 9879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:31.014128 9879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:31.015273 9879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:31.015312 9879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:31.015337 9879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:31.017045 9879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:37367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:31.017335 9879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:31.017627 9879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:31.023351 9884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:31.023360 9885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:31.023377 9879 server_base.cc:1034] running on GCE node
W20260923 22:17:31.023360 9887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:31.023839 9879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:31.024088 9879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:31.025240 9879 hybrid_clock.cc:648] HybridClock initialized: now 1790201851025218 us; error 38 us; skew 500 ppm
I20260923 22:17:31.026436 9879 webserver.cc:460] Webserver started at http://127.4.207.3:36471/ using document root <none> and password file <none>
I20260923 22:17:31.026649 9879 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:31.026695 9879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:31.026789 9879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:31.028012 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "4afcb8f9190643d4bc0658d064a89391"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "e87571f8d5821aa3546c710f6edcea0f"
server_key_iv: "0919ef72d18f84152490bfd80909cb75"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.028355 9879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "4afcb8f9190643d4bc0658d064a89391"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "e87571f8d5821aa3546c710f6edcea0f"
server_key_iv: "0919ef72d18f84152490bfd80909cb75"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.029681 9879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:31.030414 9893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:31.030606 9879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:31.030678 9879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "4afcb8f9190643d4bc0658d064a89391"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "e87571f8d5821aa3546c710f6edcea0f"
server_key_iv: "0919ef72d18f84152490bfd80909cb75"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.030730 9879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:31.034116 9879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:31.034348 9879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:31.034449 9879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:31.034616 9879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:31.034919 9879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:31.034950 9879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:31.034970 9879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:31.035017 9879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:31.040911 9879 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:39935
I20260923 22:17:31.040949 10006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:39935 every 8 connection(s)
I20260923 22:17:31.041280 9879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:31.044236 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.042009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:43791 (local address 127.4.207.62:37367)
0923 22:17:31.042079 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.042081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.042295 (+ 214us) server_negotiation.cc:407] Connection header received
0923 22:17:31.042750 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.042751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.042756 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.042818 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.043999 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.044001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.044001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.044005 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.044018 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.044022 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.044080 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.044169 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.044182 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:31.044438 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.041990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:37367 (local address 127.4.207.3:43791)
0923 22:17:31.042170 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.042216 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.042666 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.042872 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.042875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.042883 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:31.043851 (+ 968us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:31.043852 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:31.043881 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:31.043910 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:31.044078 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:31.044081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:31.044083 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:31.044163 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:17:31.044211 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260923 22:17:31.044926 10007 heartbeater.cc:347] Connected to a master server at 127.4.207.62:37367
I20260923 22:17:31.045030 10007 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:31.045192 10007 heartbeater.cc:511] Master 127.4.207.62:37367 requested a full tablet report, sending...
I20260923 22:17:31.045557 9557 ts_manager.cc:194] Registered new tserver with Master: 4afcb8f9190643d4bc0658d064a89391 (127.4.207.3:39935)
I20260923 22:17:31.047988 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 9879
I20260923 22:17:31.048070 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:31.048281 4924 external_mini_cluster.cc:1428] Setting key c25f5bd2ffa830897e465b2544f6c025
I20260923 22:17:31.049178 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:31.051522 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.050486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.4.207.62:37367)
0923 22:17:31.050552 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.050554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.050653 (+ 99us) server_negotiation.cc:407] Connection header received
0923 22:17:31.050695 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.050696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.050699 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.050750 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.051336 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.051337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.051337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.051339 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.051349 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.051352 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.051389 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.051420 (+ 31us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.051430 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:17:31.052870 9557 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:31.054316 9557 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:31.057956 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.056350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39727 (local address 127.0.0.1:55868)
0923 22:17:31.056692 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.056703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.056749 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.057027 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.057027 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.057030 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:31.057446 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:31.057446 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:31.057462 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:31.057485 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:31.057814 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:31.057815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:31.057816 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:31.057871 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:17:31.057886 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":86,"threads_started":1}
I20260923 22:17:31.057971 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.056722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.4.207.1:39727)
0923 22:17:31.056876 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.056881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.056891 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:31.056917 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.056919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.056927 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.057213 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.057548 (+ 335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.057552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.057553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.057556 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.057587 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.057725 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.057813 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.057880 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.057894 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260923 22:17:31.058248 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.056429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.4.207.2:40001)
0923 22:17:31.056591 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.056595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.056606 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:31.056631 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.056633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.056642 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.056728 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.057613 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.057620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.057621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.057626 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.057654 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.057679 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.057737 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.057884 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.057902 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260923 22:17:31.058432 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.056342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:40001 (local address 127.0.0.1:38578)
0923 22:17:31.056460 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.056497 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.056553 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.056728 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.056732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.056739 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:31.057459 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:31.057459 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:31.057468 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:31.057485 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:31.057768 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:31.057774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:31.057777 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:31.058329 (+ 552us) client_negotiation.cc:240] Negotiation successful
0923 22:17:31.058345 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260923 22:17:31.058845 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.056552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:39935 (local address 127.0.0.1:57276)
0923 22:17:31.057157 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.057166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.057204 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.057806 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.057807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.057810 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:31.058441 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:31.058442 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:31.058448 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:31.058460 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:31.058714 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:31.058715 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:31.058715 (+ 0us) client_negotiation.cc:769] Sending connection context
0923 22:17:31.058773 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:17:31.058786 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":95,"threads_started":1}
I20260923 22:17:31.058874 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.056980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57276 (local address 127.4.207.3:39935)
0923 22:17:31.057441 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.057445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.057457 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:31.057481 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.057483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.057491 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.057560 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.058551 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.058560 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.058560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.058564 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.058611 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.058635 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.058719 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.058782 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.058796 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":88,"threads_started":1}
I20260923 22:17:31.058770 9679 tablet_service.cc:1467] Processing CreateTablet for tablet 82cb00aca8b64015b2cce5393cc30395 (DEFAULT_TABLE table=test-table [id=bf280404870d43aca08947169471b2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:31.059141 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cb00aca8b64015b2cce5393cc30395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:31.059031 9810 tablet_service.cc:1467] Processing CreateTablet for tablet 82cb00aca8b64015b2cce5393cc30395 (DEFAULT_TABLE table=test-table [id=bf280404870d43aca08947169471b2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:31.059345 9810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cb00aca8b64015b2cce5393cc30395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:31.059948 9941 tablet_service.cc:1467] Processing CreateTablet for tablet 82cb00aca8b64015b2cce5393cc30395 (DEFAULT_TABLE table=test-table [id=bf280404870d43aca08947169471b2ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:31.060184 9941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82cb00aca8b64015b2cce5393cc30395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:31.062568 10021 tablet_bootstrap.cc:492] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c: Bootstrap starting.
I20260923 22:17:31.063118 10023 tablet_bootstrap.cc:492] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: Bootstrap starting.
I20260923 22:17:31.063344 10021 tablet_bootstrap.cc:654] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:31.063670 10021 log.cc:824] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:31.063776 10023 tablet_bootstrap.cc:654] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:31.064006 10022 tablet_bootstrap.cc:492] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86: Bootstrap starting.
I20260923 22:17:31.064037 10023 log.cc:824] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:31.064467 10021 tablet_bootstrap.cc:492] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c: No bootstrap required, opened a new log
I20260923 22:17:31.064625 10022 tablet_bootstrap.cc:654] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:31.064616 10021 ts_tablet_manager.cc:1397] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:17:31.064821 10023 tablet_bootstrap.cc:492] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: No bootstrap required, opened a new log
I20260923 22:17:31.064949 10022 log.cc:824] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:31.064977 10023 ts_tablet_manager.cc:1397] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:17:31.065589 10022 tablet_bootstrap.cc:492] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86: No bootstrap required, opened a new log
I20260923 22:17:31.065665 10022 ts_tablet_manager.cc:1397] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:31.067515 10022 raft_consensus.cc:348] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.067684 10022 raft_consensus.cc:374] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:31.067728 10022 raft_consensus.cc:729] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d78fab6619642caaaa33ed3f4574a86, State: Initialized, Role: FOLLOWER
I20260923 22:17:31.067703 10023 raft_consensus.cc:348] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.067703 10021 raft_consensus.cc:348] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.067874 10023 raft_consensus.cc:374] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:31.067874 10021 raft_consensus.cc:374] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:31.067873 10022 consensus_queue.cc:260] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.067919 10023 raft_consensus.cc:729] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4afcb8f9190643d4bc0658d064a89391, State: Initialized, Role: FOLLOWER
I20260923 22:17:31.067919 10021 raft_consensus.cc:729] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f37b59f8ca6482ab711fd81e43f9a4c, State: Initialized, Role: FOLLOWER
I20260923 22:17:31.068032 10021 consensus_queue.cc:260] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.068043 10023 consensus_queue.cc:260] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.068173 10022 ts_tablet_manager.cc:1428] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:17:31.068281 9745 heartbeater.cc:503] Master 127.4.207.62:37367 was elected leader, sending a full tablet report...
I20260923 22:17:31.068287 10021 ts_tablet_manager.cc:1428] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 22:17:31.068287 10023 ts_tablet_manager.cc:1428] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 22:17:31.068388 9876 heartbeater.cc:503] Master 127.4.207.62:37367 was elected leader, sending a full tablet report...
I20260923 22:17:31.068409 10007 heartbeater.cc:503] Master 127.4.207.62:37367 was elected leader, sending a full tablet report...
W20260923 22:17:31.177320 9877 tablet.cc:2369] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:31.204085 10029 raft_consensus.cc:482] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:31.204247 10029 raft_consensus.cc:504] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.204654 10029 leader_election.cc:288] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f37b59f8ca6482ab711fd81e43f9a4c (127.4.207.2:40001), 2d78fab6619642caaaa33ed3f4574a86 (127.4.207.1:39727)
I20260923 22:17:31.206094 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.204923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:56735 (local address 127.4.207.2:40001)
0923 22:17:31.204978 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.204980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.204994 (+ 14us) server_negotiation.cc:407] Connection header received
0923 22:17:31.205090 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.205091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.205097 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.205173 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.205808 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.205809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.205809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.205813 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.205849 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.205854 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.205919 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.206006 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.206020 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:17:31.206101 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.204975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:49007 (local address 127.4.207.1:39727)
0923 22:17:31.205031 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.205033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.205161 (+ 128us) server_negotiation.cc:407] Connection header received
0923 22:17:31.205208 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.205209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.205214 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.205265 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.205870 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.205871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.205871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.205875 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.205889 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.205893 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.205952 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.206005 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.206020 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 22:17:31.206139 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.204846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:40001 (local address 127.4.207.3:56735)
0923 22:17:31.204947 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.204961 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.205007 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.205191 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.205192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.205194 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:31.205656 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:31.205657 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:31.205664 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:31.205705 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:31.205940 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:31.205940 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:31.205941 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:31.205997 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 22:17:31.206013 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 22:17:31.206162 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.204913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39727 (local address 127.4.207.3:49007)
0923 22:17:31.205073 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.205082 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.205129 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.205286 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.205287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.205289 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:31.205775 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:17:31.205775 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:31.205781 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:17:31.205793 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:31.205954 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:31.205955 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:31.205956 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:31.206007 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:17:31.206017 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260923 22:17:31.206535 9699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cb00aca8b64015b2cce5393cc30395" candidate_uuid: "4afcb8f9190643d4bc0658d064a89391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d78fab6619642caaaa33ed3f4574a86" is_pre_election: true
I20260923 22:17:31.206516 9830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cb00aca8b64015b2cce5393cc30395" candidate_uuid: "4afcb8f9190643d4bc0658d064a89391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" is_pre_election: true
I20260923 22:17:31.206713 9699 raft_consensus.cc:2454] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4afcb8f9190643d4bc0658d064a89391 in term 0.
I20260923 22:17:31.206749 9830 raft_consensus.cc:2454] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4afcb8f9190643d4bc0658d064a89391 in term 0.
I20260923 22:17:31.206964 9897 leader_election.cc:302] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d78fab6619642caaaa33ed3f4574a86, 4afcb8f9190643d4bc0658d064a89391; no voters:
I20260923 22:17:31.207171 10029 raft_consensus.cc:2781] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:31.207234 10029 raft_consensus.cc:482] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:31.207252 10029 raft_consensus.cc:3037] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:31.208065 10029 raft_consensus.cc:504] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.208226 10029 leader_election.cc:288] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [CANDIDATE]: Term 1 election: Requested vote from peers 8f37b59f8ca6482ab711fd81e43f9a4c (127.4.207.2:40001), 2d78fab6619642caaaa33ed3f4574a86 (127.4.207.1:39727)
I20260923 22:17:31.208371 9830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cb00aca8b64015b2cce5393cc30395" candidate_uuid: "4afcb8f9190643d4bc0658d064a89391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c"
I20260923 22:17:31.208388 9699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82cb00aca8b64015b2cce5393cc30395" candidate_uuid: "4afcb8f9190643d4bc0658d064a89391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d78fab6619642caaaa33ed3f4574a86"
I20260923 22:17:31.208459 9830 raft_consensus.cc:3037] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:31.208458 9699 raft_consensus.cc:3037] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:31.209316 9830 raft_consensus.cc:2454] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4afcb8f9190643d4bc0658d064a89391 in term 1.
I20260923 22:17:31.209316 9699 raft_consensus.cc:2454] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4afcb8f9190643d4bc0658d064a89391 in term 1.
I20260923 22:17:31.209519 9896 leader_election.cc:302] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4afcb8f9190643d4bc0658d064a89391, 8f37b59f8ca6482ab711fd81e43f9a4c; no voters:
I20260923 22:17:31.209683 10029 raft_consensus.cc:2781] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:31.209852 10029 raft_consensus.cc:686] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 1 LEADER]: Becoming Leader. State: Replica: 4afcb8f9190643d4bc0658d064a89391, State: Running, Role: LEADER
I20260923 22:17:31.209950 10029 consensus_queue.cc:237] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } }
I20260923 22:17:31.210819 9555 catalog_manager.cc:5495] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4afcb8f9190643d4bc0658d064a89391 (127.4.207.3). New cstate: current_term: 1 leader_uuid: "4afcb8f9190643d4bc0658d064a89391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4afcb8f9190643d4bc0658d064a89391" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39935 } health_report { overall_health: HEALTHY } } }
I20260923 22:17:31.242208 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.241226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57284 (local address 127.4.207.3:39935)
0923 22:17:31.241309 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.241310 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.241318 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:31.241354 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.241354 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.241368 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.241421 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.242004 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.242005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.242006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.242010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.242025 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.242029 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.242082 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.242127 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.242139 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 22:17:31.244156 9699 raft_consensus.cc:1261] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [term 1 FOLLOWER]: Refusing update from remote peer 4afcb8f9190643d4bc0658d064a89391: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:31.244169 9830 raft_consensus.cc:1261] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 1 FOLLOWER]: Refusing update from remote peer 4afcb8f9190643d4bc0658d064a89391: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:31.244517 10031 consensus_queue.cc:1035] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d78fab6619642caaaa33ed3f4574a86" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:31.244657 10029 consensus_queue.cc:1035] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f37b59f8ca6482ab711fd81e43f9a4c" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:31.247016 10034 mvcc.cc:205] Tried to move back new op lower bound from 7332666782693289984 to 7332666782556745728. Current Snapshot: MvccSnapshot[applied={T|T < 7332666782693289984}]
I20260923 22:17:31.249493 9555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:31.249630 9555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:31.250504 9555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2954a03a6e9a430d8f3cad91f2d3499c: Sending DeleteTablet for 3 replicas of tablet 82cb00aca8b64015b2cce5393cc30395
I20260923 22:17:31.250906 9810 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cb00aca8b64015b2cce5393cc30395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:31 UTC) from {username='slave'} at 127.0.0.1:38578
I20260923 22:17:31.250914 9941 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cb00aca8b64015b2cce5393cc30395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:31 UTC) from {username='slave'} at 127.0.0.1:57276
I20260923 22:17:31.251178 9679 tablet_service.cc:1514] Processing DeleteTablet for tablet 82cb00aca8b64015b2cce5393cc30395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:31 UTC) from {username='slave'} at 127.0.0.1:55868
I20260923 22:17:31.251384 10046 tablet_replica.cc:330] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86: stopping tablet replica
I20260923 22:17:31.251490 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 9617
I20260923 22:17:31.251511 10046 raft_consensus.cc:2229] T 82cb00aca8b64015b2cce5393cc30395 P 2d78fab6619642caaaa33ed3f4574a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:31.252087 10044 tablet_replica.cc:330] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: stopping tablet replica
I20260923 22:17:31.252182 10044 raft_consensus.cc:2229] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:31.252270 10044 raft_consensus.cc:2258] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:31.252748 10044 ts_tablet_manager.cc:1905] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:31.254745 10044 ts_tablet_manager.cc:1918] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:31.254824 10044 log.cc:1196] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/82cb00aca8b64015b2cce5393cc30395
I20260923 22:17:31.255122 10044 ts_tablet_manager.cc:1939] T 82cb00aca8b64015b2cce5393cc30395 P 4afcb8f9190643d4bc0658d064a89391: Deleting consensus metadata
I20260923 22:17:31.255571 9543 catalog_manager.cc:4841] TS 4afcb8f9190643d4bc0658d064a89391 (127.4.207.3:39935): tablet 82cb00aca8b64015b2cce5393cc30395 (table test-table [id=bf280404870d43aca08947169471b2ab]) successfully deleted
W20260923 22:17:31.257911 9544 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:39727 (error 108)
I20260923 22:17:31.258496 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.258341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39727 (local address 127.0.0.1:55882)
0923 22:17:31.258406 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.258460 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 22:17:31.258538 10045 tablet_replica.cc:330] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c: stopping tablet replica
I20260923 22:17:31.258538 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 9748
I20260923 22:17:31.258665 10045 raft_consensus.cc:2229] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:31.258792 10045 raft_consensus.cc:2258] T 82cb00aca8b64015b2cce5393cc30395 P 8f37b59f8ca6482ab711fd81e43f9a4c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 22:17:31.260382 9544 catalog_manager.cc:4584] TS 2d78fab6619642caaaa33ed3f4574a86 (127.4.207.1:39727): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82cb00aca8b64015b2cce5393cc30395: Network error: Client connection negotiation failed: client connection to 127.4.207.1:39727: connect: Connection refused (error 111)
I20260923 22:17:31.264583 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 9879
I20260923 22:17:31.264635 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.264527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:40001 (local address 127.0.0.1:38586)
0923 22:17:31.264572 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.264605 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.2:40001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 22:17:31.269253 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 9526
2026-09-23T22:17:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (733 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 22:17:31.289098 4924 mini_oidc.cc:152] Starting JWKS server
I20260923 22:17:31.290141 4924 webserver.cc:460] Webserver started at https://127.0.0.1:38063/ using document root <none> and password file <none>
I20260923 22:17:31.290251 4924 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 22:17:31.290377 4924 webserver.cc:460] Webserver started at http://127.0.0.1:37193/ using document root <none> and password file <none>
2026-09-23T22:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:31Z Disabled control of system clock
I20260923 22:17:31.298343 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:46007
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:46007
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38063/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/testchainca.cert with env {}
W20260923 22:17:31.378989 10053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:31.379226 10053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:31.379251 10053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:31.380426 10053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:31.380463 10053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:31.380476 10053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:31.380488 10053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:31.380501 10053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 22:17:31.380512 10053 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 22:17:31.380529 10053 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38063/jwks.json
I20260923 22:17:31.381987 10053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:46007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:46007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38063/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:31.382215 10053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:31.382473 10053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:31.387362 10053 server_base.cc:1034] running on GCE node
W20260923 22:17:31.387346 10059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:31.387317 10058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:31.387312 10061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:31.387850 10053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:31.388193 10053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:31.389361 10053 hybrid_clock.cc:648] HybridClock initialized: now 1790201851389338 us; error 37 us; skew 500 ppm
I20260923 22:17:31.390671 10053 webserver.cc:460] Webserver started at http://127.4.207.62:40249/ using document root <none> and password file <none>
I20260923 22:17:31.390923 10053 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:31.390996 10053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:31.391157 10053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:31.392330 10053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "a012b3d28df746dfa3e4dc312f0de4b2"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "f0cf32660bbffa6f42c973fa121052e1"
server_key_iv: "81b17ece23b26da3f17f5c6f5c9fd62f"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.392738 10053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "a012b3d28df746dfa3e4dc312f0de4b2"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "f0cf32660bbffa6f42c973fa121052e1"
server_key_iv: "81b17ece23b26da3f17f5c6f5c9fd62f"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.394088 10053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:31.394799 10067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:31.394979 10053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:31.395083 10053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "a012b3d28df746dfa3e4dc312f0de4b2"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "f0cf32660bbffa6f42c973fa121052e1"
server_key_iv: "81b17ece23b26da3f17f5c6f5c9fd62f"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.395171 10053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:31.415462 10053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:31.415843 10053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:31.416013 10053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:31.420130 10053 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:46007
I20260923 22:17:31.420167 10119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:46007 every 8 connection(s)
I20260923 22:17:31.420558 10053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:31.421602 10120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:31.423070 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 10053
I20260923 22:17:31.423194 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:31.423389 4924 external_mini_cluster.cc:1428] Setting key dae5184c2195d04568e359d0383a78cb
I20260923 22:17:31.424396 10120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2: Bootstrap starting.
I20260923 22:17:31.425071 10120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:31.425417 10120 log.cc:824] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:31.426119 10120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2: No bootstrap required, opened a new log
I20260923 22:17:31.428560 10120 raft_consensus.cc:348] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012b3d28df746dfa3e4dc312f0de4b2" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46007 } }
I20260923 22:17:31.428761 10120 raft_consensus.cc:374] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:31.428790 10120 raft_consensus.cc:729] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a012b3d28df746dfa3e4dc312f0de4b2, State: Initialized, Role: FOLLOWER
I20260923 22:17:31.428926 10120 consensus_queue.cc:260] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012b3d28df746dfa3e4dc312f0de4b2" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46007 } }
I20260923 22:17:31.429008 10120 raft_consensus.cc:388] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:31.429071 10120 raft_consensus.cc:482] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:31.429123 10120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:31.429540 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.423956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45314 (local address 127.4.207.62:46007)
0923 22:17:31.424184 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.424192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.424217 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:17:31.424668 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.424676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.424740 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.424860 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.425166 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.426288 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.427401 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.427875 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.427967 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.428731 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.428734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.428734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.429009 (+ 275us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.429058 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.429079 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.429176 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.429241 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.429328 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260923 22:17:31.429776 10120 raft_consensus.cc:504] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012b3d28df746dfa3e4dc312f0de4b2" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46007 } }
I20260923 22:17:31.429884 10120 leader_election.cc:302] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a012b3d28df746dfa3e4dc312f0de4b2; no voters:
I20260923 22:17:31.430037 10120 leader_election.cc:288] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:31.430058 10125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:31.430186 10125 raft_consensus.cc:686] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [term 1 LEADER]: Becoming Leader. State: Replica: a012b3d28df746dfa3e4dc312f0de4b2, State: Running, Role: LEADER
I20260923 22:17:31.430256 10125 consensus_queue.cc:237] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012b3d28df746dfa3e4dc312f0de4b2" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46007 } }
I20260923 22:17:31.430290 10120 sys_catalog.cc:564] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:31.430460 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:31.431550 10126 sys_catalog.cc:455] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a012b3d28df746dfa3e4dc312f0de4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012b3d28df746dfa3e4dc312f0de4b2" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46007 } } }
I20260923 22:17:31.431649 10126 sys_catalog.cc:458] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:31.431821 10127 sys_catalog.cc:455] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a012b3d28df746dfa3e4dc312f0de4b2. Latest consensus state: current_term: 1 leader_uuid: "a012b3d28df746dfa3e4dc312f0de4b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012b3d28df746dfa3e4dc312f0de4b2" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46007 } } }
I20260923 22:17:31.431860 10140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:31.432137 10127 sys_catalog.cc:458] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:31.432512 10140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:31.434022 10140 catalog_manager.cc:1347] Generated new cluster ID: 1eb392c2960b40559e9d41f77f4f9753
I20260923 22:17:31.434141 10140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:31.447578 10140 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:31.448323 10140 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:31.454180 10140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a012b3d28df746dfa3e4dc312f0de4b2: Generated new TSK 0
I20260923 22:17:31.454385 10140 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 22:17:31.487691 10150 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 22:17:31.487815 10150 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 22:17:31.488045 10123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.465240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.4.207.62:46007)
0923 22:17:31.465305 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.465308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.465459 (+ 151us) server_negotiation.cc:407] Connection header received
0923 22:17:31.465566 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.465568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.465580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.465724 (+ 144us) server_negotiation.cc:226] Negotiated authn=JWT
0923 22:17:31.465909 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.466410 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.467204 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.467415 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.467525 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.467763 (+ 238us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 22:17:31.487811 (+ 20048us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 22:17:31.487989 (+ 178us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45316: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 22:17:31.488137 10123 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45316: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 22:17:31.488471 10149 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:31.465186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:46007 (local address 127.0.0.1:45316)
0923 22:17:31.465386 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.465396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.465482 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.465737 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.465738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.465740 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 22:17:31.465816 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:31.465823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.466497 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.466498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:31.467107 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:31.467109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.467531 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.467533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:31.467602 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.467629 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 22:17:31.488011 (+ 20382us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 22:17:31.488132 (+ 121us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.207.62:46007: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.207.62:46007: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38063/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 22:17:31.489114 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 10053
2026-09-23T22:17:31Z chronyd exiting
I20260923 22:17:31.525429 4924 test_util.cc:175] -----------------------------------------------
I20260923 22:17:31.525559 4924 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790201827590652-4924-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 22:17:31.526342 4924 mini_oidc.cc:152] Starting JWKS server
I20260923 22:17:31.527061 4924 webserver.cc:460] Webserver started at https://127.0.0.1:40035/ using document root <none> and password file <none>
I20260923 22:17:31.527132 4924 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 22:17:31.527326 4924 webserver.cc:460] Webserver started at http://127.0.0.1:37383/ using document root <none> and password file <none>
2026-09-23T22:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:31Z Disabled control of system clock
I20260923 22:17:31.535553 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:46359
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:43005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:46359
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40035/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/testchainca.cert with env {}
W20260923 22:17:31.615852 10156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:31.616076 10156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:31.616101 10156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:31.617304 10156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:31.617350 10156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:31.617363 10156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:31.617376 10156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:31.617435 10156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 22:17:31.617453 10156 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 22:17:31.617478 10156 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40035/jwks.json
I20260923 22:17:31.619040 10156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:46359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40035/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:31.619325 10156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:31.619576 10156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:31.624899 10156 server_base.cc:1034] running on GCE node
W20260923 22:17:31.624785 10164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:31.624787 10161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:31.624786 10162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:31.626310 10156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:31.626623 10156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:31.627818 10156 hybrid_clock.cc:648] HybridClock initialized: now 1790201851627785 us; error 65 us; skew 500 ppm
I20260923 22:17:31.629267 10156 webserver.cc:460] Webserver started at http://127.4.207.62:44515/ using document root <none> and password file <none>
I20260923 22:17:31.629562 10156 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:31.629648 10156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:31.629815 10156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:31.631021 10156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "da678ca9e7714f47b5a70c580703e07d"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "16d31792539882a8a0cc094e0afcba5b"
server_key_iv: "60ecfacda2cbf2c5d2fc24b42ea25619"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.631429 10156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "da678ca9e7714f47b5a70c580703e07d"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "16d31792539882a8a0cc094e0afcba5b"
server_key_iv: "60ecfacda2cbf2c5d2fc24b42ea25619"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.632967 10156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:31.633874 10170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:31.634074 10156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:31.634196 10156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "da678ca9e7714f47b5a70c580703e07d"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "16d31792539882a8a0cc094e0afcba5b"
server_key_iv: "60ecfacda2cbf2c5d2fc24b42ea25619"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.634292 10156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:31.645015 10156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:31.645396 10156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:31.645571 10156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:31.649924 10156 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:46359
I20260923 22:17:31.649979 10222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:46359 every 8 connection(s)
I20260923 22:17:31.650368 10156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:31.651319 10223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:31.653645 10223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d: Bootstrap starting.
I20260923 22:17:31.654188 10223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:31.654422 10223 log.cc:824] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:31.655218 10223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d: No bootstrap required, opened a new log
I20260923 22:17:31.657488 10223 raft_consensus.cc:348] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da678ca9e7714f47b5a70c580703e07d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46359 } }
I20260923 22:17:31.657644 10223 raft_consensus.cc:374] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:31.657668 10223 raft_consensus.cc:729] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da678ca9e7714f47b5a70c580703e07d, State: Initialized, Role: FOLLOWER
I20260923 22:17:31.657877 10223 consensus_queue.cc:260] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da678ca9e7714f47b5a70c580703e07d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46359 } }
I20260923 22:17:31.657965 10223 raft_consensus.cc:388] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:31.658012 10223 raft_consensus.cc:482] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:31.658064 10223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:31.658668 10223 raft_consensus.cc:504] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da678ca9e7714f47b5a70c580703e07d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46359 } }
I20260923 22:17:31.658756 10223 leader_election.cc:302] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da678ca9e7714f47b5a70c580703e07d; no voters:
I20260923 22:17:31.658883 10223 leader_election.cc:288] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:31.658973 10226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:31.659129 10226 raft_consensus.cc:686] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [term 1 LEADER]: Becoming Leader. State: Replica: da678ca9e7714f47b5a70c580703e07d, State: Running, Role: LEADER
I20260923 22:17:31.659145 10223 sys_catalog.cc:564] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:31.659222 10226 consensus_queue.cc:237] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da678ca9e7714f47b5a70c580703e07d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46359 } }
I20260923 22:17:31.659637 10228 sys_catalog.cc:455] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [sys.catalog]: SysCatalogTable state changed. Reason: New leader da678ca9e7714f47b5a70c580703e07d. Latest consensus state: current_term: 1 leader_uuid: "da678ca9e7714f47b5a70c580703e07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da678ca9e7714f47b5a70c580703e07d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46359 } } }
I20260923 22:17:31.659982 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 10156
I20260923 22:17:31.660087 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:31.660307 4924 external_mini_cluster.cc:1428] Setting key 3cf93db879b2a8828ae6236420d69071
I20260923 22:17:31.659751 10227 sys_catalog.cc:455] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da678ca9e7714f47b5a70c580703e07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da678ca9e7714f47b5a70c580703e07d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 46359 } } }
I20260923 22:17:31.660583 10227 sys_catalog.cc:458] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:31.660709 10228 sys_catalog.cc:458] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d [sys.catalog]: This master's current role is: LEADER
W20260923 22:17:31.661085 10241 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:31.661154 10241 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:31.661206 10243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:31.661357 10243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:31.662891 10243 catalog_manager.cc:1347] Generated new cluster ID: fbe15ed15a314438ba375a4c3e43ef98
I20260923 22:17:31.662945 10243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:31.665742 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.660984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.4.207.62:46359)
0923 22:17:31.661185 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.661196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.661234 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:17:31.661362 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.661368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.661458 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.661585 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.661840 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.662887 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.663995 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.664397 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.664487 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.665089 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.665093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.665093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.665265 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.665298 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.665308 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.665360 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.665436 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.665523 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260923 22:17:31.666646 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:31.678131 10243 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:31.678680 10243 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:31.682263 10243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da678ca9e7714f47b5a70c580703e07d: Generated new TSK 0
I20260923 22:17:31.682475 10243 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 22:17:31.710570 10253 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 22:17:31.710687 10253 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 22:17:31.710999 10245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.696118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49786 (local address 127.4.207.62:46359)
0923 22:17:31.696169 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.696171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.696246 (+ 75us) server_negotiation.cc:407] Connection header received
0923 22:17:31.696322 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.696324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.696347 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.696422 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0923 22:17:31.696594 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.697402 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.698226 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.698555 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.698636 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.698845 (+ 209us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 22:17:31.710763 (+ 11918us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 22:17:31.710925 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260923 22:17:31.711061 10245 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49786: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 22:17:31.711084 10252 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:31.696006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:46359 (local address 127.0.0.1:49786)
0923 22:17:31.696163 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.696175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.696247 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.696453 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.696454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.696457 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0923 22:17:31.696512 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:31.696514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.697511 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.697512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:31.698137 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:31.698139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.698655 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.698656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:31.698721 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.698726 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 22:17:31.710882 (+ 12156us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 22:17:31.710986 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.207.62:46359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260923 22:17:31.711274 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 10156
2026-09-23T22:17:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 22:17:31.748528 4924 mini_oidc.cc:152] Starting JWKS server
I20260923 22:17:31.749248 4924 webserver.cc:460] Webserver started at https://127.0.0.1:37125/ using document root <none> and password file <none>
I20260923 22:17:31.749323 4924 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 22:17:31.749485 4924 webserver.cc:460] Webserver started at http://127.0.0.1:43507/ using document root <none> and password file <none>
2026-09-23T22:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:31Z Disabled control of system clock
I20260923 22:17:31.757325 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33039
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:33039
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37125/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 22:17:31.837162 10259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:31.837371 10259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:31.837392 10259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:31.838490 10259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:31.838524 10259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:31.838537 10259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:31.838549 10259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:31.838562 10259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 22:17:31.838573 10259 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 22:17:31.838585 10259 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37125/jwks.json
I20260923 22:17:31.839900 10259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:33039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37125/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:31.840204 10259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:31.840549 10259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:31.845368 10265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:31.845389 10267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:31.845368 10264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:31.845546 10259 server_base.cc:1034] running on GCE node
I20260923 22:17:31.845805 10259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:31.846086 10259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:31.847299 10259 hybrid_clock.cc:648] HybridClock initialized: now 1790201851847285 us; error 29 us; skew 500 ppm
I20260923 22:17:31.852067 10259 webserver.cc:460] Webserver started at http://127.4.207.62:45931/ using document root <none> and password file <none>
I20260923 22:17:31.852335 10259 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:31.852384 10259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:31.852555 10259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:31.853724 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "0a33de5ede344ca1a3c0f717a00c317e"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "b36360bd834336d60b0f1de0f34b60a5"
server_key_iv: "77dcdedb051c883eb645469e59959091"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.854100 10259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "0a33de5ede344ca1a3c0f717a00c317e"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "b36360bd834336d60b0f1de0f34b60a5"
server_key_iv: "77dcdedb051c883eb645469e59959091"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.855501 10259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:31.856220 10273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:31.856395 10259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:31.856500 10259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "0a33de5ede344ca1a3c0f717a00c317e"
format_stamp: "Formatted at 2026-09-23 22:17:31 on dist-test-slave-7511"
server_key: "b36360bd834336d60b0f1de0f34b60a5"
server_key_iv: "77dcdedb051c883eb645469e59959091"
server_key_version: "encryptionkey@0"
I20260923 22:17:31.856577 10259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:31.881357 10259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:31.881723 10259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:31.881891 10259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:31.885800 10259 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:33039
I20260923 22:17:31.885836 10325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:33039 every 8 connection(s)
I20260923 22:17:31.886210 10259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:31.886883 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:31.889700 10326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e: Bootstrap starting.
I20260923 22:17:31.890300 10326 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:31.890586 10326 log.cc:824] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:31.891314 10326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e: No bootstrap required, opened a new log
I20260923 22:17:31.891904 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 10259
I20260923 22:17:31.892015 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:31.892195 4924 external_mini_cluster.cc:1428] Setting key 99494a97a9691cfc212537cad9614a8f
I20260923 22:17:31.893572 10326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a33de5ede344ca1a3c0f717a00c317e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33039 } }
I20260923 22:17:31.893733 10326 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:31.893785 10326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a33de5ede344ca1a3c0f717a00c317e, State: Initialized, Role: FOLLOWER
I20260923 22:17:31.893973 10326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a33de5ede344ca1a3c0f717a00c317e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33039 } }
I20260923 22:17:31.894084 10331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:31.894177 10331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a33de5ede344ca1a3c0f717a00c317e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33039 } }
I20260923 22:17:31.894313 10326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:31.894371 10326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:31.894313 10331 leader_election.cc:302] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a33de5ede344ca1a3c0f717a00c317e; no voters:
I20260923 22:17:31.894403 10326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:31.894528 10331 leader_election.cc:288] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260923 22:17:31.895040 10326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a33de5ede344ca1a3c0f717a00c317e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33039 } }
I20260923 22:17:31.895133 10326 leader_election.cc:302] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a33de5ede344ca1a3c0f717a00c317e; no voters:
I20260923 22:17:31.895175 10326 leader_election.cc:288] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:31.895179 10332 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260923 22:17:31.895313 10332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:31.895347 10332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [term 1 LEADER]: Becoming Leader. State: Replica: 0a33de5ede344ca1a3c0f717a00c317e, State: Running, Role: LEADER
I20260923 22:17:31.895413 10332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a33de5ede344ca1a3c0f717a00c317e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33039 } }
I20260923 22:17:31.895484 10326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:31.895767 10331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a33de5ede344ca1a3c0f717a00c317e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a33de5ede344ca1a3c0f717a00c317e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33039 } } }
I20260923 22:17:31.895884 10331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:31.896092 10333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a33de5ede344ca1a3c0f717a00c317e. Latest consensus state: current_term: 1 leader_uuid: "0a33de5ede344ca1a3c0f717a00c317e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a33de5ede344ca1a3c0f717a00c317e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33039 } } }
I20260923 22:17:31.896405 10333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:31.896595 10338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:31.897119 10338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:31.898941 10338 catalog_manager.cc:1347] Generated new cluster ID: 5a84a6b8e23b4e2fb0ba03abda115d91
I20260923 22:17:31.898993 10338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:31.899057 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.892705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.4.207.62:33039)
0923 22:17:31.892918 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.892925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.892950 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:17:31.893575 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.893589 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.893661 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.893823 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:31.894054 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.895601 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.896713 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.897381 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.897513 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.898183 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:31.898187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:31.898187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:31.898401 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:31.898445 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:31.898460 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:31.898551 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:31.898635 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 22:17:31.898801 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260923 22:17:31.903986 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:31.909628 10338 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:31.910151 10338 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:31.917902 10338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a33de5ede344ca1a3c0f717a00c317e: Generated new TSK 0
I20260923 22:17:31.918125 10338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:31.929657 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:31.925829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.4.207.62:33039)
0923 22:17:31.925905 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:31.925907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:31.926078 (+ 171us) server_negotiation.cc:407] Connection header received
0923 22:17:31.926190 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:31.926191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:31.926206 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:31.926286 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0923 22:17:31.926452 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.926973 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.927889 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.928107 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.928201 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.928395 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 22:17:31.929410 (+ 1015us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37125/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 22:17:31.929573 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37125/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 22:17:31.929723 10330 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37125/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 22:17:31.929723 10355 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:31.925775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33039 (local address 127.0.0.1:51594)
0923 22:17:31.925990 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:31.926001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:31.926099 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:31.926288 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:31.926289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:31.926291 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 22:17:31.926355 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:31.926358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.927073 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.927074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:31.927766 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:31.927769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:31.928203 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:31.928204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:31.928270 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:31.928276 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 22:17:31.929521 (+ 1245us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37125/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 22:17:31.929626 (+ 105us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.207.62:33039: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37125/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":148,"threads_started":1}
I20260923 22:17:31.929872 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 10259
E20260923 22:17:31.940558 10356 webserver.cc:533] Webserver: sslize failed:
E20260923 22:17:31.940665 10356 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T22:17:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T22:17:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:31Z Disabled control of system clock
I20260923 22:17:31.974704 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33521
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:32867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:33521
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/testchainca.cert with env {}
W20260923 22:17:32.055696 10360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:32.055946 10360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:32.055985 10360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:32.057207 10360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:32.057281 10360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:32.057301 10360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:32.057322 10360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:32.057353 10360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 22:17:32.057379 10360 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 22:17:32.057400 10360 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 22:17:32.059093 10360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:32867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:33521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:32.059393 10360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:32.059675 10360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:32.064703 10366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:32.064889 10365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:32.064736 10368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:32.065377 10360 server_base.cc:1034] running on GCE node
I20260923 22:17:32.065654 10360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:32.066015 10360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:32.067174 10360 hybrid_clock.cc:648] HybridClock initialized: now 1790201852067134 us; error 52 us; skew 500 ppm
I20260923 22:17:32.068910 10360 webserver.cc:460] Webserver started at http://127.4.207.62:37741/ using document root <none> and password file <none>
I20260923 22:17:32.069226 10360 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:32.069295 10360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:32.069437 10360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:32.070605 10360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "2ce354ffcb354de38532536866fd3903"
format_stamp: "Formatted at 2026-09-23 22:17:32 on dist-test-slave-7511"
server_key: "6384ea81d27c271d8607ac3b8693996e"
server_key_iv: "455697439a40be30fa5e713d5717bf84"
server_key_version: "encryptionkey@0"
I20260923 22:17:32.070987 10360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "2ce354ffcb354de38532536866fd3903"
format_stamp: "Formatted at 2026-09-23 22:17:32 on dist-test-slave-7511"
server_key: "6384ea81d27c271d8607ac3b8693996e"
server_key_iv: "455697439a40be30fa5e713d5717bf84"
server_key_version: "encryptionkey@0"
I20260923 22:17:32.072422 10360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:32.073230 10374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:32.073438 10360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:32.073539 10360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "2ce354ffcb354de38532536866fd3903"
format_stamp: "Formatted at 2026-09-23 22:17:32 on dist-test-slave-7511"
server_key: "6384ea81d27c271d8607ac3b8693996e"
server_key_iv: "455697439a40be30fa5e713d5717bf84"
server_key_version: "encryptionkey@0"
I20260923 22:17:32.073603 10360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:32.096462 10360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:32.096766 10360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:32.096872 10360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:32.101035 10360 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:33521
I20260923 22:17:32.101071 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:33521 every 8 connection(s)
I20260923 22:17:32.101473 10360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:32.102414 10427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:32.104756 10427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903: Bootstrap starting.
I20260923 22:17:32.105343 10427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:32.105638 10427 log.cc:824] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:32.106331 10427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903: No bootstrap required, opened a new log
I20260923 22:17:32.107995 10427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce354ffcb354de38532536866fd3903" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33521 } }
I20260923 22:17:32.108150 10427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:32.108172 10427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce354ffcb354de38532536866fd3903, State: Initialized, Role: FOLLOWER
I20260923 22:17:32.108345 10427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce354ffcb354de38532536866fd3903" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33521 } }
I20260923 22:17:32.108407 10427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:32.108428 10427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:32.108454 10427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:32.109025 10427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce354ffcb354de38532536866fd3903" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33521 } }
I20260923 22:17:32.109146 10427 leader_election.cc:302] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ce354ffcb354de38532536866fd3903; no voters:
I20260923 22:17:32.109202 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 10360
I20260923 22:17:32.109345 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:32.109349 10427 leader_election.cc:288] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:32.109376 10430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:32.109495 10430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [term 1 LEADER]: Becoming Leader. State: Replica: 2ce354ffcb354de38532536866fd3903, State: Running, Role: LEADER
I20260923 22:17:32.109557 4924 external_mini_cluster.cc:1428] Setting key 49aec0abf8560d37ac2d8611acb9b344
I20260923 22:17:32.109584 10430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce354ffcb354de38532536866fd3903" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33521 } }
I20260923 22:17:32.109629 10427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:32.109925 10430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ce354ffcb354de38532536866fd3903. Latest consensus state: current_term: 1 leader_uuid: "2ce354ffcb354de38532536866fd3903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce354ffcb354de38532536866fd3903" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33521 } } }
I20260923 22:17:32.109989 10430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:32.110301 10431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ce354ffcb354de38532536866fd3903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce354ffcb354de38532536866fd3903" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33521 } } }
I20260923 22:17:32.110433 10431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:32.110458 10436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:32.111061 10436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:32.112887 10436 catalog_manager.cc:1347] Generated new cluster ID: ceb484318e5e4eb8a09de39500c884a4
I20260923 22:17:32.112946 10436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:32.116163 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.110033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.4.207.62:33521)
0923 22:17:32.110623 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:32.110632 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:32.110692 (+ 60us) server_negotiation.cc:407] Connection header received
0923 22:17:32.110812 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:32.110819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:32.110890 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:32.111342 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:32.111471 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.112718 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.113856 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.114449 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.114882 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:32.115308 (+ 426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:32.115312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:32.115312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:17:32.115533 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:32.115570 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:17:32.115583 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:32.115856 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:32.115887 (+ 31us) server_negotiation.cc:299] Negotiation successful
0923 22:17:32.115963 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":88,"threads_started":1}
I20260923 22:17:32.117115 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:17:32.127310 10436 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:32.128145 10436 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:32.133667 10436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ce354ffcb354de38532536866fd3903: Generated new TSK 0
I20260923 22:17:32.133844 10436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:32.205909 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:32.141083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.4.207.62:33521)
0923 22:17:32.141135 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:32.141138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:32.141241 (+ 103us) server_negotiation.cc:407] Connection header received
0923 22:17:32.141321 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:32.141322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:32.141334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:32.141402 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0923 22:17:32.141569 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.142088 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.143219 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.143421 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.143541 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:32.143716 (+ 175us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 22:17:32.205664 (+ 61948us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 22:17:32.205843 (+ 179us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45728: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":28}
W20260923 22:17:32.206007 10438 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45728: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 22:17:32.206096 10456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:32.140969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33521 (local address 127.0.0.1:45728)
0923 22:17:32.141158 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:32.141170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:32.141246 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:32.141419 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:32.141419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:32.141421 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 22:17:32.141495 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:32.141497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.142173 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.142174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:32.143127 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:32.143131 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:32.143536 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:32.143538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:32.143611 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:32.143618 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 22:17:32.205826 (+ 62208us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 22:17:32.205975 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.207.62:33521: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":120,"threads_started":1}
I20260923 22:17:32.206447 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 10360
2026-09-23T22:17:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (256 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:32 dist-test-slave-7511 krb5kdc[10468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:32 dist-test-slave-7511 krb5kdc[10468](info): set up 2 sockets
Sep 23 22:17:32 dist-test-slave-7511 krb5kdc[10468](info): commencing operation
krb5kdc: starting...
W20260923 22:17:33.439776 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:33 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:33.562590 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39053
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:39053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:33.643428 10481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:33.643714 10481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:33.643754 10481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:33.644928 10481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:33.645007 10481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:33.645030 10481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:33.645051 10481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:33.645081 10481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:33.646560 10481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:39053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:33.646850 10481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:33.647082 10481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:33.652035 10487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:33.652046 10481 server_base.cc:1034] running on GCE node
W20260923 22:17:33.652014 10489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:33.652004 10486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:33.652524 10481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:33.652853 10481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:33.654019 10481 hybrid_clock.cc:648] HybridClock initialized: now 1790201853654014 us; error 35 us; skew 500 ppm
Sep 23 22:17:33 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:33.656539 10481 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:33.656939 10481 webserver.cc:460] Webserver started at http://127.4.207.62:34715/ using document root <none> and password file <none>
I20260923 22:17:33.657130 10481 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:33.657171 10481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:33.657259 10481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:33.658263 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "10f9e80b3a4442098cfce438940c280d"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "a351145986a6bf3a2cab2b87216d774a"
server_key_iv: "387965071106040f9763825b3ccde617"
server_key_version: "encryptionkey@0"
I20260923 22:17:33.658564 10481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "10f9e80b3a4442098cfce438940c280d"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "a351145986a6bf3a2cab2b87216d774a"
server_key_iv: "387965071106040f9763825b3ccde617"
server_key_version: "encryptionkey@0"
I20260923 22:17:33.659898 10481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:33.660632 10496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:33.660808 10481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:33.660898 10481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "10f9e80b3a4442098cfce438940c280d"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "a351145986a6bf3a2cab2b87216d774a"
server_key_iv: "387965071106040f9763825b3ccde617"
server_key_version: "encryptionkey@0"
I20260923 22:17:33.660974 10481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:33.674600 10481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:33.677210 10481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:33.677373 10481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:33.681360 10481 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:39053
I20260923 22:17:33.681417 10548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:39053 every 8 connection(s)
I20260923 22:17:33.681810 10481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:33.682687 10549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:33.685055 10549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d: Bootstrap starting.
I20260923 22:17:33.685635 10549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:33.685923 10549 log.cc:824] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:33.686583 10549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d: No bootstrap required, opened a new log
I20260923 22:17:33.687428 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 10481
I20260923 22:17:33.687579 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:33.687769 4924 external_mini_cluster.cc:1428] Setting key 897b3e73ac8c9510068101ad0b475d60
I20260923 22:17:33.688568 10549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f9e80b3a4442098cfce438940c280d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39053 } }
I20260923 22:17:33.688736 10549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:33.688782 10549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f9e80b3a4442098cfce438940c280d, State: Initialized, Role: FOLLOWER
I20260923 22:17:33.688958 10549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f9e80b3a4442098cfce438940c280d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39053 } }
I20260923 22:17:33.689056 10549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:33.689102 10549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:33.689152 10549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:33.689855 10549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f9e80b3a4442098cfce438940c280d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39053 } }
I20260923 22:17:33.689944 10549 leader_election.cc:302] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10f9e80b3a4442098cfce438940c280d; no voters:
I20260923 22:17:33.690160 10549 leader_election.cc:288] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:33.690217 10554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:33.690344 10554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [term 1 LEADER]: Becoming Leader. State: Replica: 10f9e80b3a4442098cfce438940c280d, State: Running, Role: LEADER
I20260923 22:17:33.690421 10549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:33.690428 10554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f9e80b3a4442098cfce438940c280d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39053 } }
I20260923 22:17:33.690764 10554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10f9e80b3a4442098cfce438940c280d. Latest consensus state: current_term: 1 leader_uuid: "10f9e80b3a4442098cfce438940c280d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f9e80b3a4442098cfce438940c280d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39053 } } }
I20260923 22:17:33.690834 10554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:33.690775 10555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10f9e80b3a4442098cfce438940c280d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10f9e80b3a4442098cfce438940c280d" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39053 } } }
I20260923 22:17:33.690994 10555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:33.691120 10559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:33.691630 10559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:33.693316 10559 catalog_manager.cc:1347] Generated new cluster ID: 7aa2d084d8224f699411369cab3078d0
I20260923 22:17:33.693372 10559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:33 dist-test-slave-7511 krb5kdc[10468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:33.699221 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:33.688335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.4.207.62:39053)
0923 22:17:33.688553 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:33.688561 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:33.688586 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:17:33.688686 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:33.688692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:33.688734 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:33.688902 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:33.689694 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:33.691035 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:33.692169 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:33.692636 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:33.692763 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:33.695585 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:33.695589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:33.695590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:33.695602 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:33.697569 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:33.697994 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:33.697997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:33.697997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:33.698112 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:33.698328 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:33.698329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:33.698329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:33.698608 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:33.698766 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:33.698881 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 22:17:33.699012 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":112,"threads_started":1}
I20260923 22:17:33.700465 10559 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:33.701148 10559 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:33.711858 10559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10f9e80b3a4442098cfce438940c280d: Generated new TSK 0
I20260923 22:17:33.712062 10559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:33.744485 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39053
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:33.826349 10575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:33.826577 10575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:33.826610 10575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:33.827946 10575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:33.827993 10575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:33.828022 10575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:33.829545 10575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:33.829803 10575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:33.830061 10575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:33.835227 10583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:33.835412 10575 server_base.cc:1034] running on GCE node
W20260923 22:17:33.835189 10581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:33.835218 10580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:33.835757 10575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:33.836025 10575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:33.837190 10575 hybrid_clock.cc:648] HybridClock initialized: now 1790201853837171 us; error 39 us; skew 500 ppm
Sep 23 22:17:33 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:33.839759 10575 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:33.840261 10575 webserver.cc:460] Webserver started at http://127.4.207.1:40431/ using document root <none> and password file <none>
I20260923 22:17:33.840497 10575 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:33.840564 10575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:33.840708 10575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:33.841791 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "42210f0b489e48a895b4b59d4fb85054"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "c45cc5ec3837b2ae40e022ead80428a0"
server_key_iv: "07828bcdaf43669cbf7a2f4ac16ffa23"
server_key_version: "encryptionkey@0"
I20260923 22:17:33.842172 10575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "42210f0b489e48a895b4b59d4fb85054"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "c45cc5ec3837b2ae40e022ead80428a0"
server_key_iv: "07828bcdaf43669cbf7a2f4ac16ffa23"
server_key_version: "encryptionkey@0"
I20260923 22:17:33.843576 10575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:33.844348 10590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:33.844527 10575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:33.844642 10575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "42210f0b489e48a895b4b59d4fb85054"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "c45cc5ec3837b2ae40e022ead80428a0"
server_key_iv: "07828bcdaf43669cbf7a2f4ac16ffa23"
server_key_version: "encryptionkey@0"
I20260923 22:17:33.844734 10575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:33.860018 10575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:33.862396 10575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:33.862676 10575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:33.862991 10575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:33.863354 10575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:33.863416 10575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:33.863477 10575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:33.863505 10575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:33.869591 10575 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:39205
I20260923 22:17:33.869714 10703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:39205 every 8 connection(s)
I20260923 22:17:33.869997 10575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:33 dist-test-slave-7511 krb5kdc[10468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:33.879343 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:33.870835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:42617 (local address 127.4.207.62:39053)
0923 22:17:33.870884 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:33.870886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:33.871033 (+ 147us) server_negotiation.cc:407] Connection header received
0923 22:17:33.872055 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:33.872057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:33.872071 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:33.872144 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:33.872785 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:33.873287 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:33.874265 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:33.874471 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:33.874526 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:33.876710 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:33.876711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:33.876712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:33.876726 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:33.878080 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:33.878503 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:33.878504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:33.878504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:33.878571 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:33.878800 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:33.878801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:33.878801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:33.878909 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:33.878961 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:33.879203 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 22:17:33.879274 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:33.879580 10706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:33.870754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39053 (local address 127.4.207.1:42617)
0923 22:17:33.870913 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:33.870958 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:33.871937 (+ 979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:33.872205 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:33.872209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:33.872502 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:33.872704 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:33.872709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:33.873370 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:33.873371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:33.874188 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:33.874190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:33.874547 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:33.874548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:33.874649 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:33.875148 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:33.875156 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:33.876610 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:33.878185 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:33.878192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:33.878194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:33.878425 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:33.878633 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:33.878633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:33.878634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:33.878742 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:33.878965 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:33.878970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:33.879090 (+ 120us) client_negotiation.cc:769] Sending connection context
0923 22:17:33.879174 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:17:33.879321 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260923 22:17:33.879927 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 10575
I20260923 22:17:33.880026 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:33.880234 4924 external_mini_cluster.cc:1428] Setting key ee76efc6121d98846aca08c0f22e028a
I20260923 22:17:33.880276 10704 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39053
I20260923 22:17:33.880384 10704 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:33.880584 10704 heartbeater.cc:511] Master 127.4.207.62:39053 requested a full tablet report, sending...
I20260923 22:17:33.881109 10513 ts_manager.cc:194] Registered new tserver with Master: 42210f0b489e48a895b4b59d4fb85054 (127.4.207.1:39205)
I20260923 22:17:33.881735 10513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:42617
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:33.907006 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39053
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:33.988883 10709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:33.989109 10709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:33.989133 10709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:33.990248 10709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:33.990284 10709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:33.990310 10709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:33.991997 10709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:33.992215 10709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:33.992413 10709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:33.997478 10715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:33.997478 10717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:33.997591 10709 server_base.cc:1034] running on GCE node
W20260923 22:17:33.997524 10714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:33.997972 10709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:33.998241 10709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:33.999383 10709 hybrid_clock.cc:648] HybridClock initialized: now 1790201853999373 us; error 29 us; skew 500 ppm
Sep 23 22:17:33 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.001995 10709 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:34.002429 10709 webserver.cc:460] Webserver started at http://127.4.207.2:38573/ using document root <none> and password file <none>
I20260923 22:17:34.002676 10709 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:34.002720 10709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:34.002851 10709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:34.003974 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "3f81a4ab3048407eaca81e630b703473"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "bc40eb704f8e01c1f51c347789efbff7"
server_key_iv: "872071be632832b3cc454c187c4e562a"
server_key_version: "encryptionkey@0"
I20260923 22:17:34.004331 10709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "3f81a4ab3048407eaca81e630b703473"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "bc40eb704f8e01c1f51c347789efbff7"
server_key_iv: "872071be632832b3cc454c187c4e562a"
server_key_version: "encryptionkey@0"
I20260923 22:17:34.005686 10709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:34.006421 10724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:34.006635 10709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:34.006727 10709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "3f81a4ab3048407eaca81e630b703473"
format_stamp: "Formatted at 2026-09-23 22:17:33 on dist-test-slave-7511"
server_key: "bc40eb704f8e01c1f51c347789efbff7"
server_key_iv: "872071be632832b3cc454c187c4e562a"
server_key_version: "encryptionkey@0"
I20260923 22:17:34.006800 10709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:34.028743 10709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:34.030709 10709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:34.030864 10709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:34.031119 10709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:34.031528 10709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:34.031584 10709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:34.031636 10709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:34.031664 10709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:34.037457 10709 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:35061
I20260923 22:17:34.037509 10837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:35061 every 8 connection(s)
I20260923 22:17:34.037829 10709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:34.042445 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 10709
I20260923 22:17:34.042562 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:34.042757 4924 external_mini_cluster.cc:1428] Setting key 966ac15a65a42bebdf361e5da3c595dd
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:34.047960 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.038590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:36299 (local address 127.4.207.62:39053)
0923 22:17:34.038658 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.038661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.038894 (+ 233us) server_negotiation.cc:407] Connection header received
0923 22:17:34.039837 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.039838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.039849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.039907 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:34.040516 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.041017 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.041982 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.042183 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.042236 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.044922 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:34.044924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:34.044924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:34.044932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:34.046381 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.046886 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.046887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.046887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.046935 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.047252 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.047253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.047253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.047357 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:34.047446 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.047760 (+ 314us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.047866 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:17:34.048161 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.038575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39053 (local address 127.4.207.2:36299)
0923 22:17:34.038760 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.038797 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:34.039738 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:34.039973 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:34.039976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:34.040237 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:34.040427 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.040431 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.041094 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.041094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.041910 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.041912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.042271 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.042272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.042423 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.043014 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:34.043023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:34.044816 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:34.046469 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:34.046475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:34.046478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:34.046810 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:34.047001 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:34.047002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:34.047002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:34.047165 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:34.047482 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:34.047488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:34.047620 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 22:17:34.047741 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 22:17:34.047915 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260923 22:17:34.048691 10838 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39053
I20260923 22:17:34.048775 10838 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:34.048976 10838 heartbeater.cc:511] Master 127.4.207.62:39053 requested a full tablet report, sending...
I20260923 22:17:34.049400 10513 ts_manager.cc:194] Registered new tserver with Master: 3f81a4ab3048407eaca81e630b703473 (127.4.207.2:35061)
I20260923 22:17:34.049922 10513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:36299
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:34.070952 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39053
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:34.154227 10843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:34.154458 10843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:34.154484 10843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:34.155738 10843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:34.155782 10843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:34.155809 10843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:34.157266 10843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:34.157472 10843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:34.157661 10843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:34.162963 10843 server_base.cc:1034] running on GCE node
W20260923 22:17:34.162834 10849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:34.162838 10848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:34.163098 10851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:34.163386 10843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:34.163671 10843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:34.164853 10843 hybrid_clock.cc:648] HybridClock initialized: now 1790201854164840 us; error 27 us; skew 500 ppm
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.167389 10843 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:34.167953 10843 webserver.cc:460] Webserver started at http://127.4.207.3:37195/ using document root <none> and password file <none>
I20260923 22:17:34.168195 10843 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:34.168242 10843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:34.168391 10843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:34.169472 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "d3561b00c2d24abca8c612cf0af144fd"
format_stamp: "Formatted at 2026-09-23 22:17:34 on dist-test-slave-7511"
server_key: "e4e72f85a8de8149126bbd53b4b4569f"
server_key_iv: "dad1a62e5261079820dc6d7fe14a6a90"
server_key_version: "encryptionkey@0"
I20260923 22:17:34.169838 10843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "d3561b00c2d24abca8c612cf0af144fd"
format_stamp: "Formatted at 2026-09-23 22:17:34 on dist-test-slave-7511"
server_key: "e4e72f85a8de8149126bbd53b4b4569f"
server_key_iv: "dad1a62e5261079820dc6d7fe14a6a90"
server_key_version: "encryptionkey@0"
I20260923 22:17:34.171164 10843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:17:34.171983 10858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:34.172217 10843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:34.172318 10843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "d3561b00c2d24abca8c612cf0af144fd"
format_stamp: "Formatted at 2026-09-23 22:17:34 on dist-test-slave-7511"
server_key: "e4e72f85a8de8149126bbd53b4b4569f"
server_key_iv: "dad1a62e5261079820dc6d7fe14a6a90"
server_key_version: "encryptionkey@0"
I20260923 22:17:34.172390 10843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:34.185464 10843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:34.187671 10843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:34.187848 10843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:34.188095 10843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:34.188429 10843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:34.188462 10843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:34.188506 10843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:34.188534 10843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:34.194729 10843 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:36451
I20260923 22:17:34.194803 10971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:36451 every 8 connection(s)
I20260923 22:17:34.195147 10843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:34.196058 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 10843
I20260923 22:17:34.196189 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:34.196385 4924 external_mini_cluster.cc:1428] Setting key cecd05af82f4ab63384197799e9e7cb5
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:34.205391 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.195930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:41543 (local address 127.4.207.62:39053)
0923 22:17:34.196026 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.196029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.196263 (+ 234us) server_negotiation.cc:407] Connection header received
0923 22:17:34.197513 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.197514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.197526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.197572 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:34.198234 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.198747 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.199896 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.200117 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.200206 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.202413 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:34.202415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:34.202416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:34.202424 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:34.204083 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.204519 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.204520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.204520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.204574 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.204778 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.204779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.204779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.204886 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:34.204936 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.205199 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.205265 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 22:17:34.205502 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.195939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39053 (local address 127.4.207.3:41543)
0923 22:17:34.196115 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:34.196161 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:34.197414 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:34.197655 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:34.197659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:34.197940 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:34.198140 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.198144 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.198848 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.198849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.199807 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:34.199810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.200216 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.200217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:34.200355 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.200925 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:34.200934 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:34.202312 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:34.204196 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:34.204211 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:34.204212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:34.204451 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:34.204635 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:34.204635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:34.204636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:34.204726 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:34.204926 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:34.204929 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:34.205046 (+ 117us) client_negotiation.cc:769] Sending connection context
0923 22:17:34.205153 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 22:17:34.205297 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260923 22:17:34.206000 10972 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39053
I20260923 22:17:34.206218 10972 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:34.206431 10972 heartbeater.cc:511] Master 127.4.207.62:39053 requested a full tablet report, sending...
I20260923 22:17:34.206837 10513 ts_manager.cc:194] Registered new tserver with Master: d3561b00c2d24abca8c612cf0af144fd (127.4.207.3:36451)
I20260923 22:17:34.207231 10513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:41543
I20260923 22:17:34.208207 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/test-admin.keytab.
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.225484 4924 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 22:17:34.227250 10979 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:34.226436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49488 (local address 127.4.207.1:39205)
0923 22:17:34.226610 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.226615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.226627 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:34.226653 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.226655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.226669 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.226749 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:34.227136 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49488: BlockingRecv error: recv got EOF from 127.0.0.1:49488 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
W20260923 22:17:34.227427 4924 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5cc' of type FILE
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.228518 4924 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.229501 4924 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:34.237599 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.230073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.4.207.1:39205)
0923 22:17:34.230115 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.230117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.230145 (+ 28us) server_negotiation.cc:407] Connection header received
0923 22:17:34.230205 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.230206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.230216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.230265 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:34.230654 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.231404 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.232293 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.232637 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.232695 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.234813 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:34.234821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:34.234821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:34.234828 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:34.236458 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.236817 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.236821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.236822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.236884 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.237118 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.237118 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.237119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.237279 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:34.237387 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.237456 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.237521 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 22:17:34.239665 10979 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:34.238963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49504 (local address 127.4.207.1:39205)
0923 22:17:34.239020 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.239022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.239079 (+ 57us) server_negotiation.cc:407] Connection header received
0923 22:17:34.239151 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.239152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.239169 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.239282 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:34.239594 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49504: BlockingRecv error: recv got EOF from 127.0.0.1:49504 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260923 22:17:34.239856 4924 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5cc)
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.240823 4924 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.241689 4924 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:34.248749 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.242194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49516 (local address 127.4.207.1:39205)
0923 22:17:34.242252 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.242254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.242271 (+ 17us) server_negotiation.cc:407] Connection header received
0923 22:17:34.242351 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.242352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.242359 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.242405 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:34.242747 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.243242 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.243993 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.244212 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.244271 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.246327 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:34.246328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:34.246328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:34.246335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:34.247690 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.248005 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.248005 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.248006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.248052 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.248243 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.248243 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.248244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.248332 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:34.248404 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.248533 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.248591 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 22:17:34.250560 10979 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:34.249903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.4.207.1:39205)
0923 22:17:34.249939 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.249941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.250092 (+ 151us) server_negotiation.cc:407] Connection header received
0923 22:17:34.250146 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.250146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.250154 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.250208 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:34.250523 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49522: BlockingRecv error: recv got EOF from 127.0.0.1:49522 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 22:17:34.250698 4924 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790201827590652-4924-0/krb5kdc/krb5cc' of type FILE
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.251854 4924 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:34.252758 4924 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:17:34.259837 10979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:34.253339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.4.207.1:39205)
0923 22:17:34.253408 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:34.253410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:34.253468 (+ 58us) server_negotiation.cc:407] Connection header received
0923 22:17:34.253496 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:34.253496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:34.253504 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:34.253570 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:34.253953 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.254410 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.255094 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:34.255276 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:34.255327 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:34.257376 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:34.257377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:34.257378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:34.257384 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:34.258832 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.259124 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.259124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.259125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.259172 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:34.259328 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:34.259328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:34.259329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:34.259414 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:34.259630 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:34.259715 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 22:17:34.259772 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:34.260497 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 10575
I20260923 22:17:34.265861 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 10709
I20260923 22:17:34.270191 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 10843
I20260923 22:17:34.274235 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 10481
2026-09-23T22:17:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1893 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10992](info): set up 2 sockets
Sep 23 22:17:34 dist-test-slave-7511 krb5kdc[10992](info): commencing operation
krb5kdc: starting...
W20260923 22:17:35.313329 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:35 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.62.keytab.
Entry for principal oryx/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.62.keytab.
I20260923 22:17:35.441771 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:41877
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.62.keytab
--principal=oryx/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:41877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:35.520741 11005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:35.520963 11005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:35.520983 11005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:35.522086 11005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:35.522127 11005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:35.522142 11005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:35.522154 11005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:35.522166 11005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:35.523701 11005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:41877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.62.keytab
--principal=oryx/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:35.523939 11005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:35.524195 11005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:35.529011 11013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.529014 11010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.529062 11011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:35.529201 11005 server_base.cc:1034] running on GCE node
I20260923 22:17:35.529462 11005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:35.529701 11005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:35.530838 11005 hybrid_clock.cc:648] HybridClock initialized: now 1790201855530827 us; error 28 us; skew 500 ppm
Sep 23 22:17:35 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:35.533281 11005 init.cc:377] Logged in from keytab as oryx/127.4.207.62@KRBTEST.COM (short username oryx)
I20260923 22:17:35.533768 11005 webserver.cc:460] Webserver started at http://127.4.207.62:36315/ using document root <none> and password file <none>
I20260923 22:17:35.534010 11005 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:35.534076 11005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:35.534209 11005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:35.535233 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "88824b9d36574c1a8bae6e1da0e97a81"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "6b20ff2b406387a27e8bbeae0f4ab9cf"
server_key_iv: "2453d27227c818d94e8676153536d845"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.535601 11005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "88824b9d36574c1a8bae6e1da0e97a81"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "6b20ff2b406387a27e8bbeae0f4ab9cf"
server_key_iv: "2453d27227c818d94e8676153536d845"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.536832 11005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:35.537541 11020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.537724 11005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:35.537818 11005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "88824b9d36574c1a8bae6e1da0e97a81"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "6b20ff2b406387a27e8bbeae0f4ab9cf"
server_key_iv: "2453d27227c818d94e8676153536d845"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.537921 11005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:35.553373 11005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:35.555153 11005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:35.555320 11005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:35.559257 11005 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:41877
I20260923 22:17:35.559293 11072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:41877 every 8 connection(s)
I20260923 22:17:35.559680 11005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:35.560664 11073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:35.562893 11073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: Bootstrap starting.
I20260923 22:17:35.563516 11073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:35.563812 11073 log.cc:824] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:35.564504 11073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: No bootstrap required, opened a new log
I20260923 22:17:35.566043 11073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } }
I20260923 22:17:35.566202 11073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:35.566228 11073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88824b9d36574c1a8bae6e1da0e97a81, State: Initialized, Role: FOLLOWER
I20260923 22:17:35.566442 11073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } }
I20260923 22:17:35.566517 11073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:35.566540 11073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:35.566552 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 11005
I20260923 22:17:35.566601 11073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:35.566665 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:35.566845 4924 external_mini_cluster.cc:1428] Setting key 410ad5016a49ad8854a19484256093e5
I20260923 22:17:35.567719 11073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } }
I20260923 22:17:35.567839 11073 leader_election.cc:302] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88824b9d36574c1a8bae6e1da0e97a81; no voters:
I20260923 22:17:35.567988 11073 leader_election.cc:288] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:35.568022 11078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:35.568181 11078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 1 LEADER]: Becoming Leader. State: Replica: 88824b9d36574c1a8bae6e1da0e97a81, State: Running, Role: LEADER
I20260923 22:17:35.568241 11073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:35.568297 11078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } }
I20260923 22:17:35.568635 11078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88824b9d36574c1a8bae6e1da0e97a81. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } }
I20260923 22:17:35.568715 11078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:35.568642 11079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } }
I20260923 22:17:35.568918 11079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:35.569087 11085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:35.569626 11085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:35.571334 11085 catalog_manager.cc:1347] Generated new cluster ID: 9639e6723cfd4048a92e3666e08b6bf7
I20260923 22:17:35.571590 11085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:35 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:35.577355 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.567372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.4.207.62:41877)
0923 22:17:35.567590 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:35.567600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:35.567635 (+ 35us) server_negotiation.cc:407] Connection header received
0923 22:17:35.567759 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:35.567766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:35.567839 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:35.568001 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:35.568913 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.570206 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.571171 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.571768 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.571891 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.574111 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:35.574114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:35.574115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:35.574126 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:35.575923 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.576291 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.576294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.576295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.576389 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.576592 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.576594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.576594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.576822 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:35.576996 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:35.577072 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 22:17:35.577207 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260923 22:17:35.594310 11085 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:35.595036 11085 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:35.601778 11085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: Generated new TSK 0
I20260923 22:17:35.601963 11085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab.
Entry for principal oryx/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab.
I20260923 22:17:35.630340 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41877
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab
--principal=oryx/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:35.712776 11099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:35.712983 11099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:35.713006 11099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:35.714064 11099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:35.714102 11099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:35.714128 11099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:35.715579 11099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab
--principal=oryx/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:35.715771 11099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:35.715942 11099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:35.720777 11104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.720834 11107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.720831 11105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:35.720983 11099 server_base.cc:1034] running on GCE node
I20260923 22:17:35.721210 11099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:35.721467 11099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:35.722627 11099 hybrid_clock.cc:648] HybridClock initialized: now 1790201855722620 us; error 37 us; skew 500 ppm
Sep 23 22:17:35 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:35.725226 11099 init.cc:377] Logged in from keytab as oryx/127.4.207.1@KRBTEST.COM (short username oryx)
I20260923 22:17:35.725746 11099 webserver.cc:460] Webserver started at http://127.4.207.1:40693/ using document root <none> and password file <none>
I20260923 22:17:35.725987 11099 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:35.726055 11099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:35.726192 11099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:35.727222 11099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "8b651870caa74b9ba700c170af10218c"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "80e95000e2c05b94c5456fd4bf6dc33f"
server_key_iv: "0ef8112d142444aef1277a8337cb9972"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.727599 11099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "8b651870caa74b9ba700c170af10218c"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "80e95000e2c05b94c5456fd4bf6dc33f"
server_key_iv: "0ef8112d142444aef1277a8337cb9972"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.728935 11099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:35.729667 11114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.729857 11099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:35.729951 11099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "8b651870caa74b9ba700c170af10218c"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "80e95000e2c05b94c5456fd4bf6dc33f"
server_key_iv: "0ef8112d142444aef1277a8337cb9972"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.730037 11099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:35.774056 11099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:35.776511 11099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:35.776688 11099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:35.776942 11099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:35.777261 11099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:35.777315 11099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.777359 11099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:35.777388 11099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.783174 11099 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:41353
I20260923 22:17:35.783223 11227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:41353 every 8 connection(s)
I20260923 22:17:35.783573 11099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:35.785730 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 11099
I20260923 22:17:35.785854 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:35.786036 4924 external_mini_cluster.cc:1428] Setting key aac37a2ac8ea71beef6f45fe9547e915
Sep 23 22:17:35 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:35.794063 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.784300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:34787 (local address 127.4.207.62:41877)
0923 22:17:35.784341 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:35.784343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:35.784575 (+ 232us) server_negotiation.cc:407] Connection header received
0923 22:17:35.785056 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:35.785057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:35.785069 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:35.785130 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:35.786176 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.786882 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.787988 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.788202 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.788288 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.790843 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:35.790844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:35.790845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:35.790853 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:35.792423 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.792971 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.792972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.792972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.793028 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.793347 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.793347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.793347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.793456 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:35.793507 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:35.793874 (+ 367us) server_negotiation.cc:299] Negotiation successful
0923 22:17:35.793970 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:17:35.794312 11230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.784272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.1:34787)
0923 22:17:35.784439 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:35.784483 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:35.784974 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:35.785188 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:35.785191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:35.785792 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:35.786062 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.786067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.786969 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.786969 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.787877 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.787880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.788298 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.788299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.788432 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.789020 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:35.789029 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:35.790682 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:35.792514 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.792523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.792527 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.792873 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.793112 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.793113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.793113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.793272 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.793524 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:35.793531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:35.793712 (+ 181us) client_negotiation.cc:769] Sending connection context
0923 22:17:35.793843 (+ 131us) client_negotiation.cc:240] Negotiation successful
0923 22:17:35.794026 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260923 22:17:35.794831 11228 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41877
WARNING: no policy specified for oryx/127.4.207.2@KRBTEST.COM; defaulting to no policy
I20260923 22:17:35.794948 11228 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:35.795145 11228 heartbeater.cc:511] Master 127.4.207.62:41877 requested a full tablet report, sending...
I20260923 22:17:35.795681 11037 ts_manager.cc:194] Registered new tserver with Master: 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353)
I20260923 22:17:35.796310 11037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:34787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab.
Entry for principal oryx/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab.
I20260923 22:17:35.813812 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41877
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab
--principal=oryx/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:35.894877 11233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:35.895088 11233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:35.895112 11233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:35.896317 11233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:35.896365 11233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:35.896392 11233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:35.897866 11233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab
--principal=oryx/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:35.898139 11233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:35.898415 11233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:35.903458 11239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.903496 11241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:35.903469 11238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:35.903707 11233 server_base.cc:1034] running on GCE node
I20260923 22:17:35.903900 11233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:35.904157 11233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:35.905325 11233 hybrid_clock.cc:648] HybridClock initialized: now 1790201855905325 us; error 41 us; skew 500 ppm
Sep 23 22:17:35 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:35.907847 11233 init.cc:377] Logged in from keytab as oryx/127.4.207.2@KRBTEST.COM (short username oryx)
I20260923 22:17:35.908337 11233 webserver.cc:460] Webserver started at http://127.4.207.2:33707/ using document root <none> and password file <none>
I20260923 22:17:35.908542 11233 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:35.908582 11233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:35.908675 11233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:35.909678 11233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "d3706851c2474a1cb29162189eeb077d"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "ba991e157ce68757ae577132f0a92497"
server_key_iv: "858b21bce482bc291febd28a464d6edd"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.909968 11233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "d3706851c2474a1cb29162189eeb077d"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "ba991e157ce68757ae577132f0a92497"
server_key_iv: "858b21bce482bc291febd28a464d6edd"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.911252 11233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:35.912010 11248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.912196 11233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.912292 11233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "d3706851c2474a1cb29162189eeb077d"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "ba991e157ce68757ae577132f0a92497"
server_key_iv: "858b21bce482bc291febd28a464d6edd"
server_key_version: "encryptionkey@0"
I20260923 22:17:35.912351 11233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:35.934989 11233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:35.937557 11233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:35.937683 11233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:35.937879 11233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:35.938220 11233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:35.938256 11233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.938280 11233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:35.938331 11233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:35.944439 11233 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:33925
I20260923 22:17:35.944494 11361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:33925 every 8 connection(s)
I20260923 22:17:35.944816 11233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:35.948793 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 11233
I20260923 22:17:35.948912 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:35.949095 4924 external_mini_cluster.cc:1428] Setting key 90b3343f56ccad7d847d5b18da830ebd
Sep 23 22:17:35 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:35.954892 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.945616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:47657 (local address 127.4.207.62:41877)
0923 22:17:35.945664 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:35.945666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:35.945901 (+ 235us) server_negotiation.cc:407] Connection header received
0923 22:17:35.946458 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:35.946459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:35.946471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:35.946518 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:35.947643 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.948164 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.949335 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.949595 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.949676 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.951985 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:35.951987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:35.951987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:35.951995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:35.953463 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.953902 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.953904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.953905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.953968 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:35.954231 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:35.954232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:35.954233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:35.954353 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:35.954431 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:35.954707 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 22:17:35.954800 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:35.955104 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:35.945587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.2:47657)
0923 22:17:35.945776 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:35.945813 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:35.946337 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:35.946601 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:35.946604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:35.947264 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:35.947538 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.947543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.948262 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.948263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.949230 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:35.949233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:35.949753 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:35.949754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:35.949875 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:35.950441 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:35.950450 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:35.951882 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:35.953560 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.953567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.953570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.953820 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.954038 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:35.954039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:35.954039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:35.954157 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:35.954422 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:35.954428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:35.954576 (+ 148us) client_negotiation.cc:769] Sending connection context
0923 22:17:35.954687 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 22:17:35.954849 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260923 22:17:35.955669 11362 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41877
I20260923 22:17:35.955789 11362 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:35.955974 11362 heartbeater.cc:511] Master 127.4.207.62:41877 requested a full tablet report, sending...
I20260923 22:17:35.956398 11037 ts_manager.cc:194] Registered new tserver with Master: d3706851c2474a1cb29162189eeb077d (127.4.207.2:33925)
I20260923 22:17:35.956789 11037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:47657
WARNING: no policy specified for oryx/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab.
Entry for principal oryx/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab.
I20260923 22:17:35.975286 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41877
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab
--principal=oryx/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:36.056609 11367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:36.056846 11367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:36.056885 11367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:36.058142 11367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:36.058202 11367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:36.058255 11367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:36.060907 11367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab
--principal=oryx/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:36.061198 11367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:36.061479 11367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:36.067857 11372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.067870 11373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.067885 11375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.067888 11367 server_base.cc:1034] running on GCE node
I20260923 22:17:36.068248 11367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:36.068512 11367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:36.069651 11367 hybrid_clock.cc:648] HybridClock initialized: now 1790201856069639 us; error 29 us; skew 500 ppm
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:36.072291 11367 init.cc:377] Logged in from keytab as oryx/127.4.207.3@KRBTEST.COM (short username oryx)
I20260923 22:17:36.072788 11367 webserver.cc:460] Webserver started at http://127.4.207.3:34107/ using document root <none> and password file <none>
I20260923 22:17:36.073014 11367 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.073057 11367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.073210 11367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:36.074301 11367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "6cc23253566941d4b30895e230e765ae"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "676e8adaf1a5cc4340846ca35c0930ab"
server_key_iv: "3bac75c50d329e8e9ee34be41b39cc82"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.074663 11367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "6cc23253566941d4b30895e230e765ae"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "676e8adaf1a5cc4340846ca35c0930ab"
server_key_iv: "3bac75c50d329e8e9ee34be41b39cc82"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.076048 11367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:36.076750 11382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.076931 11367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:36.077019 11367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "6cc23253566941d4b30895e230e765ae"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "676e8adaf1a5cc4340846ca35c0930ab"
server_key_iv: "3bac75c50d329e8e9ee34be41b39cc82"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.077092 11367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.113734 11367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:36.116035 11367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:36.116209 11367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:36.116451 11367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:36.116777 11367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:36.116811 11367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.116832 11367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:36.116878 11367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.122666 11367 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:38899
I20260923 22:17:36.122742 11495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:38899 every 8 connection(s)
I20260923 22:17:36.123044 11367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:36.130748 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 11367
I20260923 22:17:36.130869 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:36.131057 4924 external_mini_cluster.cc:1428] Setting key 4d44a0f0db8fe6696aae468976231a81
I20260923 22:17:36.132733 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.123864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:56797 (local address 127.4.207.62:41877)
0923 22:17:36.123929 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.123931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.124121 (+ 190us) server_negotiation.cc:407] Connection header received
0923 22:17:36.124602 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.124603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.124613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.124672 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.125554 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.126019 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.127108 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.127303 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.127354 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.129548 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.129549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.129550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.129556 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.131076 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.131588 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.131589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.131589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.131651 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.131964 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.131965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.131965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.132120 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.132188 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.132534 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.132642 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 22:17:36.132944 11498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.123837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.3:56797)
0923 22:17:36.123997 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.124036 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.124520 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.124726 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.124730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.125287 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.125477 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.125481 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.126100 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.126101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.127024 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.127027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.127375 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.127376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.127534 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.128025 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.128032 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.129452 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.131176 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.131182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.131184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.131444 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.131729 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.131730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.131730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.131888 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.132216 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.132223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.132383 (+ 160us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.132511 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.132691 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260923 22:17:36.133464 11496 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41877
I20260923 22:17:36.133591 11496 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:36.133766 11496 heartbeater.cc:511] Master 127.4.207.62:41877 requested a full tablet report, sending...
I20260923 22:17:36.134136 11037 ts_manager.cc:194] Registered new tserver with Master: 6cc23253566941d4b30895e230e765ae (127.4.207.3:38899)
I20260923 22:17:36.134488 11037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:56797
I20260923 22:17:36.143023 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.207.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.61.keytab.
Entry for principal oryx/127.4.207.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.61.keytab.
I20260923 22:17:36.168659 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.61:44157
--webserver_interface=127.4.207.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.61.keytab
--principal=oryx/127.4.207.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:41877,127.4.207.61:44157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:36.251595 11501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:36.251806 11501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:36.251827 11501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:36.252991 11501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:36.253028 11501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:36.253043 11501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:36.253055 11501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:36.253067 11501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:36.254588 11501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:41877,127.4.207.61:44157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.61.keytab
--principal=oryx/127.4.207.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.61:44157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.207.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
W20260923 22:17:36.254727 11501 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.207.62:41877,127.4.207.61:44157'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 22:17:36.254855 11501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:36.255028 11501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:36.260031 11509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.260126 11501 server_base.cc:1034] running on GCE node
W20260923 22:17:36.260043 11506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.260051 11507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.260443 11501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:36.260805 11501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:36.261978 11501 hybrid_clock.cc:648] HybridClock initialized: now 1790201856261966 us; error 30 us; skew 500 ppm
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:36.265029 11501 init.cc:377] Logged in from keytab as oryx/127.4.207.61@KRBTEST.COM (short username oryx)
I20260923 22:17:36.265533 11501 webserver.cc:460] Webserver started at http://127.4.207.61:41633/ using document root <none> and password file <none>
I20260923 22:17:36.265779 11501 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.265847 11501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.266007 11501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:36.267082 11501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data/instance:
uuid: "7d129f5f04484696b5aec295fb9f1b4b"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "5daab6caa7673cf78f342ea4d2bb1cd0"
server_key_iv: "d7b31fad81fb8989d2adc38188f4e832"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.267503 11501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal/instance:
uuid: "7d129f5f04484696b5aec295fb9f1b4b"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "5daab6caa7673cf78f342ea4d2bb1cd0"
server_key_iv: "d7b31fad81fb8989d2adc38188f4e832"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.269029 11501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:36.269881 11516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.270076 11501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.270210 11501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
uuid: "7d129f5f04484696b5aec295fb9f1b4b"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "5daab6caa7673cf78f342ea4d2bb1cd0"
server_key_iv: "d7b31fad81fb8989d2adc38188f4e832"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.270287 11501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.282686 11501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:36.284847 11501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:36.285024 11501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:36.288856 11501 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.61:44157
I20260923 22:17:36.288892 11568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.61:44157 every 8 connection(s)
I20260923 22:17:36.289261 11501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
I20260923 22:17:36.290297 11569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.291914 11569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:17:36.294168 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 11501
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.61@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.207.61@KRBTEST.COM
I20260923 22:17:36.302285 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.292616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.4.207.62:41877)
0923 22:17:36.292688 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.292690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.292900 (+ 210us) server_negotiation.cc:407] Connection header received
0923 22:17:36.293405 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.293407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.293420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.293472 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.294268 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.294978 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.296342 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.296639 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.296711 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.299208 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.299210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.299210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.299218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.300860 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.301252 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.301253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.301253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.301310 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.301549 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.301550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.301551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.301710 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.301810 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.302082 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.302163 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:17:36.302697 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.292585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43028)
0923 22:17:36.292779 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.292814 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.293289 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.293555 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.293558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.293833 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.294125 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.294131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.295071 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.295071 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.296259 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.296261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.296730 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.296731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.296865 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.297374 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.297384 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.299108 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.300935 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.300941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.300943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.301157 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.301383 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.301383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.301384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.301484 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.301802 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.301808 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.301942 (+ 134us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.302062 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.302269 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":104,"threads_started":1}
I20260923 22:17:36.303354 11569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:17:36.303943 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.294665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59300 (local address 127.4.207.61:44157)
0923 22:17:36.294824 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.294829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.294977 (+ 148us) server_negotiation.cc:407] Connection header received
0923 22:17:36.295001 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.295003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.295029 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.295106 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.295873 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.296764 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.297756 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.298105 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.298188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.300616 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.300624 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.300624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.300632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.302289 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.302666 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.302669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.302669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.302733 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.302931 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.302932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.302932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.303080 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.303231 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.303807 (+ 576us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.303878 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.61@KRBTEST.COM for oryx/127.4.207.61@KRBTEST.COM
I20260923 22:17:36.310305 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.303656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.0.0.1:59302)
0923 22:17:36.303700 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.303710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.303748 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.303958 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.303959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.304107 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.304157 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.304159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.304891 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.304892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.305356 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.305357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.305739 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.305740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.305802 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.306122 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.306125 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.307187 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.309313 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.309314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.309315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.309477 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.309693 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.309693 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.309694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.309746 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.309990 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.309991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.310026 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.310096 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.310190 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 22:17:36.310392 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.303688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.4.207.61:44157)
0923 22:17:36.303846 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.303847 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.303853 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:17:36.303876 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.303876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.303884 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.303958 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.304255 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.304792 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.305445 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.305651 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.305741 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.307285 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.307287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.307287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.307293 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.309197 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.309562 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.309563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.309563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.309620 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.309814 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.309815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.309815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.309925 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.310000 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.310117 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.310245 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260923 22:17:36.311841 11569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Bootstrap starting.
I20260923 22:17:36.312479 11569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.312757 11569 log.cc:824] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:36.313576 11569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: No bootstrap required, opened a new log
I20260923 22:17:36.315124 11569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } }
I20260923 22:17:36.315251 11569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.315274 11569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d129f5f04484696b5aec295fb9f1b4b, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.315416 11569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } }
I20260923 22:17:36.315829 11569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:36.315800 11576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } } }
I20260923 22:17:36.315990 11576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:17:36.317590 11587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:36.317651 11587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:36.319235 11501 master_runner.cc:428] Detected that this master 7d129f5f04484696b5aec295fb9f1b4b is joining an existing cluster
I20260923 22:17:36.319285 11501 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.207.61:44157
I20260923 22:17:36.327404 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.320431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.4.207.62:41877)
0923 22:17:36.320476 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.320478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.320590 (+ 112us) server_negotiation.cc:407] Connection header received
0923 22:17:36.320616 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.320617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.320628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.320725 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.321165 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.321944 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.322865 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.323097 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.323157 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.324326 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.324328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.324328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.324335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.326124 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.326613 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.326614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.326614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.326668 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.326878 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.326879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.326879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.326996 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.327073 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.327230 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.327302 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:36.327600 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.320525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.0.0.1:59308)
0923 22:17:36.320653 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.320660 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.320720 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.320879 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.320880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.321045 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.321103 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.321105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.321738 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.321740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.322545 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.322546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.323087 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.323088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.323165 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.323699 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.323704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.324185 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.326312 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.326312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.326313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.326553 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.326774 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.326774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.326775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.326827 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.327086 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.327087 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.327120 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.327253 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.327509 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":163,"thread_start_us":98,"threads_started":1}
I20260923 22:17:36.327896 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.320561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.4.207.61:44157)
0923 22:17:36.320596 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.320598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.320709 (+ 111us) server_negotiation.cc:407] Connection header received
0923 22:17:36.320782 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.320783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.320802 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.320828 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.321179 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.321656 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.322643 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.322975 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.323052 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.324296 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.324297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.324297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.324304 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.326232 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.326623 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.326625 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.326625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.326694 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.326891 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.326891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.326891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.327006 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.327046 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.327222 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.327401 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":20}
I20260923 22:17:36.327417 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.320320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43042)
0923 22:17:36.320451 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.320464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.320516 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.320738 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.320739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.320980 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.321064 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.321066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.322032 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.322033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.322782 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.322783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.323208 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.323209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.323267 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.323875 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.323879 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.324228 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.326199 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.326200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.326201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.326491 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.326751 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.326752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.326752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.326812 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.327064 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.327065 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.327108 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.327190 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.327313 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.62@KRBTEST.COM for oryx/127.4.207.61@KRBTEST.COM
I20260923 22:17:36.337730 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.330158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.0.0.1:59312)
0923 22:17:36.330333 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.330377 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.330455 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.330591 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.330594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.330955 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.331015 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.331017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.331705 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.331706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.332694 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.332696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.333067 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.333068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.333161 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.333564 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.333570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.335058 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.336736 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.336743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.336745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.336937 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.337134 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.337134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.337135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.337187 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.337446 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.337451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.337494 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.337563 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.337651 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260923 22:17:36.337778 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.330257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.4.207.61:44157)
0923 22:17:36.330296 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.330298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.330448 (+ 150us) server_negotiation.cc:407] Connection header received
0923 22:17:36.330529 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.330529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.330537 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.330576 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.331087 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.331627 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.332781 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.332986 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.333048 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.335139 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.335140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.335140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.335147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.336646 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.337008 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.337010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.337010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.337068 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.337253 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.337254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.337254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.337383 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.337450 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.337607 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.337688 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:36.338474 11037 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.207.61:44157: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b"
member_type: NON_VOTER
last_known_addr {
host: "127.4.207.61"
port: 44157
}
attrs {
promote: true
}
}
dest_uuid: "88824b9d36574c1a8bae6e1da0e97a81"
cas_config_opid_index: -1
I20260923 22:17:36.338830 11037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } }
I20260923 22:17:36.339361 11597 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7d129f5f04484696b5aec295fb9f1b4b (127.4.207.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } }
I20260923 22:17:36.339650 11543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.340440 11597 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.207.61:44157
I20260923 22:17:36.340540 11543 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Refusing update from remote peer 88824b9d36574c1a8bae6e1da0e97a81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 22:17:36.340597 11597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } }
I20260923 22:17:36.340653 11501 master.cc:537] Master@127.4.207.61:44157 shutting down...
I20260923 22:17:36.340659 11597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:36.340855 11596 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:17:36.341884 11024 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260923 22:17:36.342310 11024 consensus_peers.cc:487] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 -> Peer 7d129f5f04484696b5aec295fb9f1b4b (127.4.207.61:44157): Couldn't send request to peer 7d129f5f04484696b5aec295fb9f1b4b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 22:17:36.343411 11501 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:36.343541 11501 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.343591 11501 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: stopping tablet replica
I20260923 22:17:36.356130 11501 master.cc:552] Master@127.4.207.61:44157 shutdown complete.
I20260923 22:17:36.356215 11501 master_runner.cc:305] Clearing existing system tablet
I20260923 22:17:36.356346 11501 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.356408 11501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.357041 11501 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:36.357554 11601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.357682 11501 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.357749 11501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.357792 11501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
uuid: "7d129f5f04484696b5aec295fb9f1b4b"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "5daab6caa7673cf78f342ea4d2bb1cd0"
server_key_iv: "d7b31fad81fb8989d2adc38188f4e832"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.358385 11501 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.359824 11501 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 22:17:36.359881 11501 log.cc:1196] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 22:17:36.360109 11501 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Deleting consensus metadata
I20260923 22:17:36.360241 11501 master_runner.cc:315] Copying system tablet from 127.4.207.62:41877
I20260923 22:17:36.360865 11501 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.207.62:41877
I20260923 22:17:36.367139 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.361262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.4.207.62:41877)
0923 22:17:36.361314 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.361317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.361385 (+ 68us) server_negotiation.cc:407] Connection header received
0923 22:17:36.361465 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.361466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.361477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.361533 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.362091 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.362602 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.363355 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.363633 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.363716 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.364669 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.364670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.364671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.364678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.365999 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.366349 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.366350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.366350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.366405 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.366606 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.366606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.366607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.366725 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.366802 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.366964 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.367042 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:36.367152 11607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.361158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43058)
0923 22:17:36.361328 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.361339 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.361401 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.361801 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.361802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.361946 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.362000 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.362001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.362686 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.362687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.363254 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.363256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.363720 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.363722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.363793 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.364287 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.364291 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.364577 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.366094 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.366096 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.366096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.366276 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.366477 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.366478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.366478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.366533 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.366792 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.366793 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.366838 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.366940 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.367044 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260923 22:17:36.367471 11057 tablet_copy_service.cc:140] P 88824b9d36574c1a8bae6e1da0e97a81: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7d129f5f04484696b5aec295fb9f1b4b ({username='oryx', principal='oryx/127.4.207.61@KRBTEST.COM'} at 127.0.0.1:43058)
I20260923 22:17:36.367619 11057 tablet_copy_service.cc:161] P 88824b9d36574c1a8bae6e1da0e97a81: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7d129f5f04484696b5aec295fb9f1b4b at {username='oryx', principal='oryx/127.4.207.61@KRBTEST.COM'} at 127.0.0.1:43058: session id = 7d129f5f04484696b5aec295fb9f1b4b-00000000000000000000000000000000
I20260923 22:17:36.368274 11057 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 22:17:36.368932 11501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 22:17:36.369968 11501 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: tablet copy: Starting download of 0 data blocks...
I20260923 22:17:36.370107 11501 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: tablet copy: Starting download of 1 WAL segments...
I20260923 22:17:36.370962 11501 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 22:17:36.371819 11057 tablet_copy_service.cc:342] P 88824b9d36574c1a8bae6e1da0e97a81: Request end of tablet copy session 7d129f5f04484696b5aec295fb9f1b4b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.207.61@KRBTEST.COM'} at 127.0.0.1:43058
I20260923 22:17:36.371876 11057 tablet_copy_service.cc:433] P 88824b9d36574c1a8bae6e1da0e97a81: ending tablet copy session 7d129f5f04484696b5aec295fb9f1b4b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7d129f5f04484696b5aec295fb9f1b4b
I20260923 22:17:36.374251 11501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:36.375576 11609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.375641 11611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.375579 11608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.376235 11501 server_base.cc:1034] running on GCE node
I20260923 22:17:36.376333 11501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:36.376492 11501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:36.377630 11501 hybrid_clock.cc:648] HybridClock initialized: now 1790201856377616 us; error 30 us; skew 500 ppm
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:36.378969 11501 init.cc:377] Logged in from keytab as oryx/127.4.207.61@KRBTEST.COM (short username oryx)
I20260923 22:17:36.379283 11501 webserver.cc:460] Webserver started at http://127.4.207.61:40441/ using document root <none> and password file <none>
I20260923 22:17:36.379422 11501 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.379478 11501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.380095 11501 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.380643 11618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.380755 11501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:36.380798 11501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
uuid: "7d129f5f04484696b5aec295fb9f1b4b"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "5daab6caa7673cf78f342ea4d2bb1cd0"
server_key_iv: "d7b31fad81fb8989d2adc38188f4e832"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.380857 11501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.390614 11501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:36.392264 11501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:36.395362 11501 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.61:44157
I20260923 22:17:36.395612 11670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.61:44157 every 8 connection(s)
I20260923 22:17:36.395778 11501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
I20260923 22:17:36.396613 11671 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:17:36.397071 11671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Bootstrap starting.
I20260923 22:17:36.399273 11671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:17:36.399538 11671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: Bootstrap complete.
I20260923 22:17:36.399744 11671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } }
I20260923 22:17:36.399809 11671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7d129f5f04484696b5aec295fb9f1b4b, State: Initialized, Role: LEARNER
I20260923 22:17:36.399834 11671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } }
I20260923 22:17:36.399994 11671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:36.400017 11674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } }
I20260923 22:17:36.400111 11674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:36.401367 11685 catalog_manager.cc:1259] Loaded cluster ID: 9639e6723cfd4048a92e3666e08b6bf7
I20260923 22:17:36.401413 11685 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: loading cluster ID for follower catalog manager: success
I20260923 22:17:36.402386 11685 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: acquiring CA information for follower catalog manager: success
I20260923 22:17:36.402669 11685 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:17:36.412073 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.405664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.4.207.61:44157)
0923 22:17:36.405767 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.405769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.405778 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:36.405801 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.405801 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.405819 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.405891 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.406410 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.407024 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.407779 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.408007 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.408094 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.409335 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.409336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.409336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.409343 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.410778 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.411186 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.411189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.411189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.411244 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.411468 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.411469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.411469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.411577 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.411700 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.411850 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.411932 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
WARNING: no policy specified for oryx/127.4.207.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.60.keytab.
Entry for principal oryx/127.4.207.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.60.keytab.
I20260923 22:17:36.439296 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.60:40935
--webserver_interface=127.4.207.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.60.keytab
--principal=oryx/127.4.207.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:41877,127.4.207.61:44157,127.4.207.60:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:36.523602 11689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:36.523903 11689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:36.523942 11689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:36.525170 11689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:36.525238 11689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:36.525264 11689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:36.525295 11689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:36.525321 11689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:36.526881 11689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:41877,127.4.207.61:44157,127.4.207.60:40935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.60.keytab
--principal=oryx/127.4.207.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.60:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.207.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:36.527159 11689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:36.527391 11689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:36.532317 11695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.532282 11697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.532295 11694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.532738 11689 server_base.cc:1034] running on GCE node
I20260923 22:17:36.533036 11689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:36.533392 11689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:36.534555 11689 hybrid_clock.cc:648] HybridClock initialized: now 1790201856534545 us; error 28 us; skew 500 ppm
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:36.537096 11689 init.cc:377] Logged in from keytab as oryx/127.4.207.60@KRBTEST.COM (short username oryx)
I20260923 22:17:36.537582 11689 webserver.cc:460] Webserver started at http://127.4.207.60:34709/ using document root <none> and password file <none>
I20260923 22:17:36.537781 11689 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.537824 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.537912 11689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:36.538918 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data/instance:
uuid: "8f6d32aa7811463cbd1591740ebf9638"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "9007e68d7e4e40e8235f6498a147bdae"
server_key_iv: "a26b8555902be5733be38801f31f50a6"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.539222 11689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal/instance:
uuid: "8f6d32aa7811463cbd1591740ebf9638"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "9007e68d7e4e40e8235f6498a147bdae"
server_key_iv: "a26b8555902be5733be38801f31f50a6"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.540635 11689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:36.541460 11704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.541646 11689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.541731 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
uuid: "8f6d32aa7811463cbd1591740ebf9638"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "9007e68d7e4e40e8235f6498a147bdae"
server_key_iv: "a26b8555902be5733be38801f31f50a6"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.541788 11689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.556599 11689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:36.559021 11689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:36.559175 11689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:36.564388 11689 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.60:40935
I20260923 22:17:36.564417 11756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.60:40935 every 8 connection(s)
I20260923 22:17:36.564790 11689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
I20260923 22:17:36.565785 11757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:36.567408 11757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.60@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:36.574609 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 11689
I20260923 22:17:36.576752 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.568335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.4.207.62:41877)
0923 22:17:36.568398 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.568401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.568598 (+ 197us) server_negotiation.cc:407] Connection header received
0923 22:17:36.569070 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.569071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.569081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.569128 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.569859 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.570359 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.571335 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.571601 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.571656 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.573648 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.573650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.573650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.573656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.575151 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.575652 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.575654 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.575654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.575728 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.576047 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.576048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.576048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.576157 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.576209 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.576558 (+ 349us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.576657 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:17:36.576977 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.568305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43060)
0923 22:17:36.568476 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.568514 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.568974 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.569275 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.569279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.569577 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.569775 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.569779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.570441 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.570442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.571261 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.571262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.571679 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.571679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.571783 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.572232 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.572240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.573566 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.575249 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.575256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.575260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.575563 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.575813 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.575815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.575815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.575965 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.576227 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.576233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.576402 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.576521 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.576704 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
I20260923 22:17:36.577582 11757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.207.60@KRBTEST.COM
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.60@KRBTEST.COM for oryx/127.4.207.61@KRBTEST.COM
I20260923 22:17:36.584638 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.575236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.4.207.60:40935)
0923 22:17:36.575413 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.575418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.575530 (+ 112us) server_negotiation.cc:407] Connection header received
0923 22:17:36.575596 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.575598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.575614 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.575688 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.576469 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.577544 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.578556 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.578868 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.578963 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.581718 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.581725 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.581725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.581733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.583231 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.583709 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.583713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.583713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.583771 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.583962 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.583963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.583963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.584152 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.584293 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.584401 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.584489 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":115,"threads_started":1}
I20260923 22:17:36.585214 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.577879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.0.0.1:59358)
0923 22:17:36.577993 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.578004 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.578052 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.578167 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.578167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.578308 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.578359 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.578360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.579028 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.579029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.579831 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.579834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.580234 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.580235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.580306 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.580819 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.580825 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.582226 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.584168 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.584169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.584169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.584413 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.584626 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.584626 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.584627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.584685 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.584962 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.584963 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.585014 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.585080 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.585154 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 22:17:36.585301 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.577943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.4.207.61:44157)
0923 22:17:36.577995 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.577997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.578040 (+ 43us) server_negotiation.cc:407] Connection header received
0923 22:17:36.578101 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.578102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.578114 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.578163 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.578427 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.578897 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.579924 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.580144 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.580234 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.582317 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.582318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.582318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.582326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.584085 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.584488 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.584489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.584489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.584548 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.584767 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.584769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.584769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.584878 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.584953 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.585116 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.585202 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:17:36.585763 11757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.60@KRBTEST.COM for oryx/127.4.207.60@KRBTEST.COM
I20260923 22:17:36.591868 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.586046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.0.0.1:42434)
0923 22:17:36.586100 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.586111 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.586156 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.586258 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.586258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.586374 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.586418 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.586419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.586988 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.586989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.587517 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.587519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.587950 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.587951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.587993 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.588440 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.588445 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.589543 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.590972 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.590973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.590973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.591125 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.591312 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.591313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.591313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.591361 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.591649 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.591650 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.591681 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.591734 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.591804 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 22:17:36.591989 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.586110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.4.207.60:40935)
0923 22:17:36.586160 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.586161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.586168 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:36.586201 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.586201 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.586210 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.586255 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.586473 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.586924 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.587616 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.587884 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.587937 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.589617 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.589618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.589618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.589624 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.590882 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.591198 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.591199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.591199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.591250 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.591413 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.591414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.591414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.591583 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.591642 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.591753 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.591851 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":34}
I20260923 22:17:36.593258 11757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Bootstrap starting.
I20260923 22:17:36.594256 11757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:36.594553 11757 log.cc:824] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:36.595355 11757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: No bootstrap required, opened a new log
I20260923 22:17:36.597328 11757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } }
I20260923 22:17:36.597496 11757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:36.597548 11757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6d32aa7811463cbd1591740ebf9638, State: Initialized, Role: FOLLOWER
I20260923 22:17:36.597723 11757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } }
I20260923 22:17:36.598085 11757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:36.598057 11762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } } }
I20260923 22:17:36.598181 11762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:17:36.599888 11773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:17:36.599953 11773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:17:36.602329 11689 master_runner.cc:428] Detected that this master 8f6d32aa7811463cbd1591740ebf9638 is joining an existing cluster
I20260923 22:17:36.602377 11689 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.207.60:40935
I20260923 22:17:36.627522 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.603505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.4.207.62:41877)
0923 22:17:36.603558 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.603560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.604159 (+ 599us) server_negotiation.cc:407] Connection header received
0923 22:17:36.604248 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.604249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.604261 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.604325 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.604701 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.605493 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.606630 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.606923 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.606968 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.624462 (+ 17494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.624464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.624464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.624473 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.626210 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.626669 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.626671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.626671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.626728 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.626917 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.626917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.626918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.627051 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.627092 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.627308 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.627391 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 22:17:36.627727 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.603654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.0.0.1:42440)
0923 22:17:36.604210 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.604221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.604287 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.604458 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.604458 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.604712 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.604793 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.604795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.605633 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.605634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.606177 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.606178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.606597 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.606598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.606662 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.607168 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.607178 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.624427 (+ 17249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.626539 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.626540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.626540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.626747 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.626932 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.626932 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.626933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.626981 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.627252 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.627252 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.627288 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.627349 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.627554 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":493,"threads_started":1}
I20260923 22:17:36.628034 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.603363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.4.207.60:40935)
0923 22:17:36.603417 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.603421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.604272 (+ 851us) server_negotiation.cc:407] Connection header received
0923 22:17:36.604358 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.604358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.604368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.604398 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.604876 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.605545 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.606253 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.606491 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.606574 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.624533 (+ 17959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.624535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.624535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.624542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.626449 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.626822 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.626823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.626823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.626871 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.627039 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.627039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.627040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.627175 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.627232 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.627495 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.627946 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":376,"server-negotiator.queue_time_us":23}
I20260923 22:17:36.628266 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.603773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59362 (local address 127.4.207.61:44157)
0923 22:17:36.603825 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.603828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.607616 (+ 3788us) server_negotiation.cc:407] Connection header received
0923 22:17:36.607684 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.607686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.607696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.607752 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.608088 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.608548 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.609268 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.609487 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.609570 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.624555 (+ 14985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.624556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.624556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.624564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.627057 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.627455 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.627456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.627457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.627511 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.627711 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.627712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.627712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.627860 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.627902 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.628110 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.628208 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 22:17:36.628548 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.603775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.0.0.1:59362)
0923 22:17:36.607537 (+ 3762us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.607547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.607625 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.607755 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.607756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.607943 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.608002 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.608004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.608628 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.608628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.609169 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.609171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.609571 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.609580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.609648 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.610045 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.610050 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.624431 (+ 14381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.627150 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.627151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.627151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.627356 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.627586 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.627587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.627587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.627644 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.627951 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.627951 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.627999 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.628360 (+ 361us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.628460 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3746,"thread_start_us":63,"threads_started":1}
I20260923 22:17:36.629316 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.603493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43070)
0923 22:17:36.603653 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.603665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.604188 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.604340 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.604340 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.604551 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.604619 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.604621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.605593 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.605594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.606494 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.606497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.607070 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.607071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.607132 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.607506 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.607510 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.624307 (+ 16797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.626312 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.626313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.626313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.626568 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.626799 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.626799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.626800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.626852 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.627135 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.627136 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.627200 (+ 64us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.629101 (+ 1901us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.629208 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201855, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.62@KRBTEST.COM for oryx/127.4.207.60@KRBTEST.COM
I20260923 22:17:36.636065 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 11099
I20260923 22:17:36.641839 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.631479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.0.0.1:42442)
0923 22:17:36.631535 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.631547 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.631592 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.631736 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.631737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.631900 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.631948 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.631950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.632580 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.632581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.633362 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.633365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.633761 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.633762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.633833 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.634169 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.634174 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.635548 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.640669 (+ 5121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.640670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.640670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.640895 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.641125 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.641149 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.641150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.641213 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.641491 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.641492 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.641539 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.641628 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.641739 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:17:36.641846 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.631548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42442 (local address 127.4.207.60:40935)
0923 22:17:36.631604 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.631605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.631614 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:36.631660 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.631660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.631668 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.631714 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.632016 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.632527 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.633441 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.633694 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.633748 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.635645 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.635647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.635647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.635654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.640538 (+ 4884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.640990 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.640991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.640992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.641048 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.641279 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.641280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.641280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.641437 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.641498 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.641696 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.641767 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:17:36.642616 11037 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.207.60:40935: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638"
member_type: NON_VOTER
last_known_addr {
host: "127.4.207.60"
port: 40935
}
attrs {
promote: true
}
}
dest_uuid: "88824b9d36574c1a8bae6e1da0e97a81"
cas_config_opid_index: 5
I20260923 22:17:36.642875 11037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } }
I20260923 22:17:36.643221 11597 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8f6d32aa7811463cbd1591740ebf9638 (127.4.207.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.643488 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:41353
--local_ip_for_outbound_sockets=127.4.207.1
--tserver_master_addrs=127.4.207.62:41877
--webserver_port=40693
--webserver_interface=127.4.207.1
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab
--principal=oryx/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157,127.4.207.60:40935 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:17:36.644196 11597 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.207.60:40935
I20260923 22:17:36.644290 11597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.644361 11597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:36.644972 11689 master.cc:537] Master@127.4.207.60:40935 shutting down...
I20260923 22:17:36.645170 11731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:36.646311 11731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Refusing update from remote peer 88824b9d36574c1a8bae6e1da0e97a81: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 22:17:36.646638 11782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:17:36.647320 11021 consensus_peers.cc:487] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 -> Peer 8f6d32aa7811463cbd1591740ebf9638 (127.4.207.60:40935): Couldn't send request to peer 8f6d32aa7811463cbd1591740ebf9638. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 22:17:36.648058 11689 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:36.648149 11689 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:36.648197 11689 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: stopping tablet replica
I20260923 22:17:36.649170 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.644622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.0.0.1:59364)
0923 22:17:36.645071 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.645082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.645125 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.645349 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.645350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.645351 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:36.645395 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.645396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.646378 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.646379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.647598 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.647601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.648682 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.648683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.648739 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.648758 (+ 19us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.649027 (+ 269us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.649076 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432}
I20260923 22:17:36.649500 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.644694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.4.207.61:44157)
0923 22:17:36.645038 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.645040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.645185 (+ 145us) server_negotiation.cc:407] Connection header received
0923 22:17:36.645211 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.645213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.645234 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.645288 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:36.645468 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.646255 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.647690 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.648593 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.648683 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.648767 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.649363 (+ 596us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.649409 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306}
I20260923 22:17:36.650259 11645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 LEARNER]: Refusing update from remote peer 88824b9d36574c1a8bae6e1da0e97a81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 22:17:36.650456 11782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 22:17:36.650985 11645 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8f6d32aa7811463cbd1591740ebf9638 (127.4.207.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.652304 11674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88824b9d36574c1a8bae6e1da0e97a81. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.652395 11674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:36.652429 11674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: NON_VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: true } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.652479 11674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:36.652740 11782 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: attempting to promote NON_VOTER 7d129f5f04484696b5aec295fb9f1b4b to VOTER
I20260923 22:17:36.652930 11782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } }
I20260923 22:17:36.653364 11645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 LEARNER]: Refusing update from remote peer 88824b9d36574c1a8bae6e1da0e97a81: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 22:17:36.653435 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.653266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.0.0.1:42448)
0923 22:17:36.653348 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.653399 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 22:17:36.653687 11596 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 22:17:36.653872 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.653711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.0.0.1:42458)
0923 22:17:36.653816 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.653844 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260923 22:17:36.653957 11021 consensus_peers.cc:487] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 -> Peer 8f6d32aa7811463cbd1591740ebf9638 (127.4.207.60:40935): Couldn't send request to peer 8f6d32aa7811463cbd1591740ebf9638. Status: Network error: Client connection negotiation failed: client connection to 127.4.207.60:40935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 22:17:36.654577 11597 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7d129f5f04484696b5aec295fb9f1b4b (127.4.207.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.654807 11645 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7d129f5f04484696b5aec295fb9f1b4b (127.4.207.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.655427 11597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.655525 11597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:36.655575 11597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.655612 11597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:36.655853 11785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.655954 11785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:36.661180 11689 master.cc:552] Master@127.4.207.60:40935 shutdown complete.
I20260923 22:17:36.661233 11689 master_runner.cc:305] Clearing existing system tablet
I20260923 22:17:36.661363 11689 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.661409 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.662154 11689 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:36.662712 11787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.662864 11689 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.662925 11689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.662949 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
uuid: "8f6d32aa7811463cbd1591740ebf9638"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "9007e68d7e4e40e8235f6498a147bdae"
server_key_iv: "a26b8555902be5733be38801f31f50a6"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.663875 11689 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:36.665429 11689 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 22:17:36.665500 11689 log.cc:1196] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 22:17:36.665710 11689 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Deleting consensus metadata
I20260923 22:17:36.665828 11689 master_runner.cc:315] Copying system tablet from 127.4.207.62:41877
I20260923 22:17:36.666440 11689 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: tablet copy: Beginning tablet copy session from remote peer at address 127.4.207.62:41877
I20260923 22:17:36.673936 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.666759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43086)
0923 22:17:36.666924 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.666935 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.666997 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.667160 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.667160 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.667361 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.667460 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.667462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.668492 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.668494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.669484 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.669487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.669973 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.669974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.670040 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.670563 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.670568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.670979 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.672824 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.672825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.672826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.673048 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.673264 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.673265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.673265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.673328 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.673585 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.673586 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.673634 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.673718 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.673839 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260923 22:17:36.673940 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.666824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.4.207.62:41877)
0923 22:17:36.666883 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.666885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.667001 (+ 116us) server_negotiation.cc:407] Connection header received
0923 22:17:36.667055 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.667056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.667068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.667141 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.667528 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.668408 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.669584 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.669889 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.669963 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.671071 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.671072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.671072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.671080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.672738 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.673130 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.673132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.673132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.673194 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.673395 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.673396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.673396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.673524 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.673583 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.673759 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.673854 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:17:36.674618 11057 tablet_copy_service.cc:140] P 88824b9d36574c1a8bae6e1da0e97a81: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8f6d32aa7811463cbd1591740ebf9638 ({username='oryx', principal='oryx/127.4.207.60@KRBTEST.COM'} at 127.0.0.1:43086)
I20260923 22:17:36.674693 11057 tablet_copy_service.cc:161] P 88824b9d36574c1a8bae6e1da0e97a81: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8f6d32aa7811463cbd1591740ebf9638 at {username='oryx', principal='oryx/127.4.207.60@KRBTEST.COM'} at 127.0.0.1:43086: session id = 8f6d32aa7811463cbd1591740ebf9638-00000000000000000000000000000000
I20260923 22:17:36.675264 11057 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 22:17:36.675745 11689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 22:17:36.677156 11689 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: tablet copy: Starting download of 0 data blocks...
I20260923 22:17:36.677397 11689 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: tablet copy: Starting download of 1 WAL segments...
I20260923 22:17:36.678313 11689 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 22:17:36.680537 11057 tablet_copy_service.cc:342] P 88824b9d36574c1a8bae6e1da0e97a81: Request end of tablet copy session 8f6d32aa7811463cbd1591740ebf9638-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.207.60@KRBTEST.COM'} at 127.0.0.1:43086
I20260923 22:17:36.680758 11057 tablet_copy_service.cc:433] P 88824b9d36574c1a8bae6e1da0e97a81: ending tablet copy session 8f6d32aa7811463cbd1591740ebf9638-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8f6d32aa7811463cbd1591740ebf9638
I20260923 22:17:36.684983 11689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:36.686322 11794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.688515 11797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.688598 11689 server_base.cc:1034] running on GCE node
W20260923 22:17:36.691900 11795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.692143 11689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:36.692310 11689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:36.693471 11689 hybrid_clock.cc:648] HybridClock initialized: now 1790201856693452 us; error 41 us; skew 500 ppm
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:36.695119 11689 init.cc:377] Logged in from keytab as oryx/127.4.207.60@KRBTEST.COM (short username oryx)
I20260923 22:17:36.695405 11689 webserver.cc:460] Webserver started at http://127.4.207.60:34985/ using document root <none> and password file <none>
I20260923 22:17:36.695557 11689 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.695600 11689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.696403 11689 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:36.697054 11804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.697232 11689 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:36.697289 11689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
uuid: "8f6d32aa7811463cbd1591740ebf9638"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "9007e68d7e4e40e8235f6498a147bdae"
server_key_iv: "a26b8555902be5733be38801f31f50a6"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.697356 11689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.722208 11689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:36.724148 11689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:36.727167 11689 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.60:40935
I20260923 22:17:36.727460 11856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.60:40935 every 8 connection(s)
I20260923 22:17:36.727536 11689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
I20260923 22:17:36.728644 11857 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:17:36.729302 11857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Bootstrap starting.
I20260923 22:17:36.732481 11857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:17:36.732713 11857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Bootstrap complete.
I20260923 22:17:36.732930 11857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } }
I20260923 22:17:36.733023 11857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f6d32aa7811463cbd1591740ebf9638, State: Initialized, Role: LEARNER
I20260923 22:17:36.733065 11857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } }
I20260923 22:17:36.733247 11857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:36.733249 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:36.733752 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:36.735209 11871 catalog_manager.cc:1259] Loaded cluster ID: 9639e6723cfd4048a92e3666e08b6bf7
I20260923 22:17:36.735493 11871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: loading cluster ID for follower catalog manager: success
I20260923 22:17:36.736608 11871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: acquiring CA information for follower catalog manager: success
I20260923 22:17:36.736924 11871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 22:17:36.747833 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:36.748062 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:36.748102 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:36.749323 11783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:36.749390 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:36.749436 11783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:36.750985 11783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab
--principal=oryx/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:41353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=40693
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157,127.4.207.60:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:36.751271 11783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:36.751580 11783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:36.756580 11877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.756709 11879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.756596 11876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.756829 11783 server_base.cc:1034] running on GCE node
I20260923 22:17:36.757036 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:36.757282 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:36.758450 11783 hybrid_clock.cc:648] HybridClock initialized: now 1790201856758440 us; error 33 us; skew 500 ppm
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:36.761181 11783 init.cc:377] Logged in from keytab as oryx/127.4.207.1@KRBTEST.COM (short username oryx)
I20260923 22:17:36.761690 11783 webserver.cc:460] Webserver started at http://127.4.207.1:40693/ using document root <none> and password file <none>
I20260923 22:17:36.761945 11783 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.762030 11783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.763386 11783 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.764036 11886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.764274 11783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:17:36.764370 11783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "8b651870caa74b9ba700c170af10218c"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "80e95000e2c05b94c5456fd4bf6dc33f"
server_key_iv: "0ef8112d142444aef1277a8337cb9972"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.764847 11783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.780108 11783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:36.782195 11783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:36.782306 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:36.782482 11783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:36.782809 11783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:36.782848 11783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.782904 11783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:36.782928 11783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.789134 11783 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:41353
I20260923 22:17:36.789202 11999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:41353 every 8 connection(s)
I20260923 22:17:36.789572 11783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:36.792348 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 11783
I20260923 22:17:36.792512 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 11233
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.60@KRBTEST.COM
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.61@KRBTEST.COM
I20260923 22:17:36.803244 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:33925
--local_ip_for_outbound_sockets=127.4.207.2
--tserver_master_addrs=127.4.207.62:41877
--webserver_port=33707
--webserver_interface=127.4.207.2
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab
--principal=oryx/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157,127.4.207.60:40935 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:17:36.803622 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.790334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:49673 (local address 127.4.207.60:40935)
0923 22:17:36.790445 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.790448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.790695 (+ 247us) server_negotiation.cc:407] Connection header received
0923 22:17:36.791823 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.791824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.791837 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.791919 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.792783 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.793870 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795227 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.795616 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795672 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.799916 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.799918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.799918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.799926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.801882 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802390 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.802391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.802391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.802450 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802762 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.802763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.802763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.802902 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.802946 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.803371 (+ 425us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.803505 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":56,"threads_started":1}
I20260923 22:17:36.803639 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.790828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:56125 (local address 127.4.207.62:41877)
0923 22:17:36.790875 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.790877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.790884 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:36.791979 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.791981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.791994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.792050 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.792788 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.793563 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795421 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.795742 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795786 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.798982 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.798984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.798984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.798992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.801900 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802459 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.802461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.802461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.802522 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802767 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.802768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.802769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.802914 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.802957 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.803377 (+ 420us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.803489 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:36.803805 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.790448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:33003 (local address 127.4.207.61:44157)
0923 22:17:36.790496 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.790498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.790669 (+ 171us) server_negotiation.cc:407] Connection header received
0923 22:17:36.791745 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.791746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.791758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.791823 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.792788 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.793563 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795420 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.795748 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795786 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.799916 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.799918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.799918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.799926 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.801714 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802391 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.802392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.802392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.802450 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802762 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.802763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.802763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.802912 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.802951 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.803628 (+ 677us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.803712 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:36.804267 12006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.790368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.4.207.1:49673)
0923 22:17:36.790610 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.790619 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.791651 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.791924 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.791925 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.792355 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.792678 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.792684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.793983 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.793984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.795124 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.795127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.795765 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.795949 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.796936 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.796941 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.799812 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.801960 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.801961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.801961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.802300 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.802527 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.802528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.802681 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.802996 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.803003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.803204 (+ 201us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.803718 (+ 514us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.803949 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":90,"thread_start_us":99,"threads_started":1}
I20260923 22:17:36.804476 12007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.790656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.1:56125)
0923 22:17:36.790753 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.790762 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.791651 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.792078 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.792079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.792350 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.792678 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.792684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.793702 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.793703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.795313 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.795316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.795867 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.795951 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.796723 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.796732 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.798831 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.801994 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.801995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.801996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.802371 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.802596 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.802597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.802694 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.803008 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.803009 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.803211 (+ 202us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.803720 (+ 509us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.804018 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":163,"thread_start_us":58,"threads_started":1}
I20260923 22:17:36.805016 12005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.790366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.4.207.1:33003)
0923 22:17:36.790529 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.790572 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.791631 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.791899 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.791903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.792350 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.792678 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.792684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.793702 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.793703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.795313 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.795316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.795869 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.795870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.795951 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.796823 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.796827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.799812 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.801817 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.801827 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.801830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.802300 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.802527 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.802528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.802528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.802681 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.802996 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.803003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.803202 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.804518 (+ 1316us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.804893 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":46,"thread_start_us":92,"threads_started":1}
I20260923 22:17:36.805105 12000 heartbeater.cc:347] Connected to a master server at 127.4.207.60:40935
I20260923 22:17:36.805202 12000 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:36.805410 12000 heartbeater.cc:511] Master 127.4.207.60:40935 requested a full tablet report, sending...
I20260923 22:17:36.805686 12001 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41877
I20260923 22:17:36.805737 12001 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:36.805795 12001 heartbeater.cc:511] Master 127.4.207.62:41877 requested a full tablet report, sending...
I20260923 22:17:36.805954 12002 heartbeater.cc:347] Connected to a master server at 127.4.207.61:44157
I20260923 22:17:36.805962 11821 ts_manager.cc:194] Registered new tserver with Master: 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353)
I20260923 22:17:36.805990 12002 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:36.806039 12002 heartbeater.cc:511] Master 127.4.207.61:44157 requested a full tablet report, sending...
I20260923 22:17:36.806073 11037 ts_manager.cc:194] Re-registered known tserver with Master: 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353)
I20260923 22:17:36.806384 11635 ts_manager.cc:194] Registered new tserver with Master: 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353)
I20260923 22:17:36.806581 11037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:56125
W20260923 22:17:36.888607 12008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:36.888864 12008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:36.888906 12008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:36.890215 12008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:36.890290 12008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:36.890334 12008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:36.891965 12008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab
--principal=oryx/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:33925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=33707
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157,127.4.207.60:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:36.892249 12008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:36.892494 12008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:36.897871 12013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:36.897898 12014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.897964 12008 server_base.cc:1034] running on GCE node
W20260923 22:17:36.897909 12016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:36.898348 12008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:36.898633 12008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:36.899789 12008 hybrid_clock.cc:648] HybridClock initialized: now 1790201856899777 us; error 29 us; skew 500 ppm
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:36.908257 12008 init.cc:377] Logged in from keytab as oryx/127.4.207.2@KRBTEST.COM (short username oryx)
I20260923 22:17:36.908823 12008 webserver.cc:460] Webserver started at http://127.4.207.2:33707/ using document root <none> and password file <none>
I20260923 22:17:36.909067 12008 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:36.909129 12008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:36.910601 12008 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.911280 12023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.911506 12008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:36.911597 12008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "d3706851c2474a1cb29162189eeb077d"
format_stamp: "Formatted at 2026-09-23 22:17:35 on dist-test-slave-7511"
server_key: "ba991e157ce68757ae577132f0a92497"
server_key_iv: "858b21bce482bc291febd28a464d6edd"
server_key_version: "encryptionkey@0"
I20260923 22:17:36.911950 12008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:36.921852 12008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:36.923892 12008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:36.924011 12008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:36.924211 12008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:36.924540 12008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:36.924578 12008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.924602 12008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:36.924614 12008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:36.930681 12008 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:33925
I20260923 22:17:36.930740 12136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:33925 every 8 connection(s)
I20260923 22:17:36.931140 12008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.60@KRBTEST.COM
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
Sep 23 22:17:36 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.61@KRBTEST.COM
I20260923 22:17:36.940188 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 12008
I20260923 22:17:36.940372 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 11367
I20260923 22:17:36.943339 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.931941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.2:47709)
0923 22:17:36.932123 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.932178 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.932809 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.933087 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.933091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.933897 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.933947 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.933949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.934956 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.934957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.936088 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.936091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.936526 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.936697 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.937394 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.937398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.939773 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.941909 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.941915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.941917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.942191 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.942418 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.942419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.942419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.942523 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.942815 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.942819 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.942945 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.943029 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.943168 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260923 22:17:36.943344 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.931999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:47709 (local address 127.4.207.62:41877)
0923 22:17:36.932039 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.932041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.932278 (+ 237us) server_negotiation.cc:407] Connection header received
0923 22:17:36.932942 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.932943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.932955 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.932989 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.934054 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.934842 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936191 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.936436 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936518 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.939891 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.939893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.939894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.939902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.941813 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.942270 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.942272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.942272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.942346 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.942588 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.942589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.942590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.942750 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.942822 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.943088 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.943204 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:17:36.946007 12138 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41877
I20260923 22:17:36.946118 12138 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:36.946174 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.931942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:34949 (local address 127.4.207.60:40935)
0923 22:17:36.931987 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.931990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.932277 (+ 287us) server_negotiation.cc:407] Connection header received
0923 22:17:36.932939 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.932940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.932957 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.932989 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.934952 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.935517 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936500 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.936785 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936817 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.939271 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.939272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.939273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.939281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.944728 (+ 5447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.945070 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.945071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.945072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.945137 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.945597 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.945598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.945598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.945727 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.945799 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.945971 (+ 172us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.946071 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:17:36.946332 12138 heartbeater.cc:511] Master 127.4.207.62:41877 requested a full tablet report, sending...
I20260923 22:17:36.946419 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:38899
--local_ip_for_outbound_sockets=127.4.207.3
--tserver_master_addrs=127.4.207.62:41877
--webserver_port=34107
--webserver_interface=127.4.207.3
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab
--principal=oryx/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157,127.4.207.60:40935 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:17:36.946823 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.931941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.4.207.2:34949)
0923 22:17:36.932179 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.932188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.932809 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.933087 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.933091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.933609 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.933926 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.933934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.935589 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.935590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.936411 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.936413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.936879 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.936967 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.937576 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.937581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.939166 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.944828 (+ 5662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.944828 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.944829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.944986 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.945460 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.945461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.945462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.945527 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.945801 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.945802 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.945849 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.946643 (+ 794us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.946752 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":23,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260923 22:17:36.947340 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.932344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:51741 (local address 127.4.207.61:44157)
0923 22:17:36.932391 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:36.932393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:36.932450 (+ 57us) server_negotiation.cc:407] Connection header received
0923 22:17:36.932935 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:36.932936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:36.932949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:36.933004 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:36.934027 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.934848 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936280 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.936579 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936653 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.940290 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:36.940292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:36.940292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:36.940303 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:36.946101 (+ 5798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.946564 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.946566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.946566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.946622 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:36.946818 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:36.946819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:36.946819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:36.946956 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:36.946998 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:36.947207 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 22:17:36.947276 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:36.947481 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:36.932275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.4.207.2:51741)
0923 22:17:36.932390 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:36.932399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:36.932825 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:36.933086 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:36.933090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:36.933596 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:36.933930 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.933933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.935528 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.935529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.936197 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:36.936199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:36.936651 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:36.936652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:36.936702 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:36.937377 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:36.937388 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:36.940154 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:36.946213 (+ 6059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.946214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.946215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.946471 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.946692 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:36.946693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:36.946693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:36.946751 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:36.947041 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:36.947042 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:36.947084 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 22:17:36.947164 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:17:36.947280 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":33,"thread_start_us":57,"threads_started":1}
I20260923 22:17:36.947677 11037 ts_manager.cc:194] Re-registered known tserver with Master: d3706851c2474a1cb29162189eeb077d (127.4.207.2:33925)
I20260923 22:17:36.947953 12137 heartbeater.cc:347] Connected to a master server at 127.4.207.60:40935
I20260923 22:17:36.948031 12137 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:36.948127 12137 heartbeater.cc:511] Master 127.4.207.60:40935 requested a full tablet report, sending...
I20260923 22:17:36.948128 12139 heartbeater.cc:347] Connected to a master server at 127.4.207.61:44157
I20260923 22:17:36.948220 11037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:47709
I20260923 22:17:36.948237 12139 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:36.948318 12139 heartbeater.cc:511] Master 127.4.207.61:44157 requested a full tablet report, sending...
I20260923 22:17:36.948633 11821 ts_manager.cc:194] Registered new tserver with Master: d3706851c2474a1cb29162189eeb077d (127.4.207.2:33925)
I20260923 22:17:36.948659 11635 ts_manager.cc:194] Registered new tserver with Master: d3706851c2474a1cb29162189eeb077d (127.4.207.2:33925)
W20260923 22:17:37.028996 12144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:37.029254 12144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:37.029291 12144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:37.030469 12144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:37.030541 12144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:37.030589 12144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:37.032128 12144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab
--principal=oryx/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:38899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=34107
--tserver_master_addrs=127.4.207.62:41877,127.4.207.61:44157,127.4.207.60:40935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:37.032400 12144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:37.032684 12144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:37.038025 12144 server_base.cc:1034] running on GCE node
W20260923 22:17:37.037978 12149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:37.037985 12152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:37.037978 12150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:37.039326 12144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:37.039611 12144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:37.040750 12144 hybrid_clock.cc:648] HybridClock initialized: now 1790201857040731 us; error 38 us; skew 500 ppm
Sep 23 22:17:37 dist-test-slave-7511 krb5kdc[10992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:37.043251 12144 init.cc:377] Logged in from keytab as oryx/127.4.207.3@KRBTEST.COM (short username oryx)
I20260923 22:17:37.043759 12144 webserver.cc:460] Webserver started at http://127.4.207.3:34107/ using document root <none> and password file <none>
I20260923 22:17:37.043962 12144 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:37.044016 12144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:37.045275 12144 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:37.045830 12159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:37.046003 12144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:37.046087 12144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "6cc23253566941d4b30895e230e765ae"
format_stamp: "Formatted at 2026-09-23 22:17:36 on dist-test-slave-7511"
server_key: "676e8adaf1a5cc4340846ca35c0930ab"
server_key_iv: "3bac75c50d329e8e9ee34be41b39cc82"
server_key_version: "encryptionkey@0"
I20260923 22:17:37.046417 12144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:37.062492 12144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:37.064657 12144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:37.064777 12144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:37.064975 12144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:37.065308 12144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:37.065343 12144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:37.065366 12144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:37.065555 12144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:37.071762 12144 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:38899
I20260923 22:17:37.071839 12272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:38899 every 8 connection(s)
I20260923 22:17:37.072214 12144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:17:37 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
Sep 23 22:17:37 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.60@KRBTEST.COM
Sep 23 22:17:37 dist-test-slave-7511 krb5kdc[10992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.61@KRBTEST.COM
I20260923 22:17:37.082181 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 12144
I20260923 22:17:37.083745 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.073378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:58123 (local address 127.4.207.62:41877)
0923 22:17:37.073430 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.073432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.073523 (+ 91us) server_negotiation.cc:407] Connection header received
0923 22:17:37.074508 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.074510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.074529 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.074594 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.075360 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.075923 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.077410 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.077617 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.077675 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.080221 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.080222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.080222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.080228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.082021 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.082611 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.082612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.082612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.082672 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.082964 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.082964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.082965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.083106 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.083148 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.083522 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.083626 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:37.084280 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.073290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.3:58123)
0923 22:17:37.073476 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.073485 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.074331 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.074611 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.074612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.075031 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:37.075269 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.075274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.076042 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.076043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.077324 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.077326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.077701 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.077702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.077841 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.078417 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:37.078427 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:37.080119 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:37.082127 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:37.082134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:37.082137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:37.082492 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:37.082748 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:37.082748 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:37.082749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:37.082885 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:37.083200 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:37.083206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:37.083327 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.083838 (+ 511us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.084046 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":8,"thread_start_us":56,"threads_started":1}
I20260923 22:17:37.084555 12279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.072977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.4.207.3:52683)
0923 22:17:37.073409 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.073419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.074329 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.074714 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.074715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.075025 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:37.075270 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.075276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.076323 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.076323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.077324 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.077326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.077993 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.077994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.078073 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.078541 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:37.078546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:37.081163 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:37.083107 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:37.083108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:37.083109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:37.083340 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:37.083583 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:37.083584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:37.083584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:37.083641 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:37.084212 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:37.084213 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:37.084254 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.084335 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.084431 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":14,"thread_start_us":100,"threads_started":1}
I20260923 22:17:37.085136 12274 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41877
I20260923 22:17:37.085242 12274 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:37.085484 12274 heartbeater.cc:511] Master 127.4.207.62:41877 requested a full tablet report, sending...
I20260923 22:17:37.085741 12278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.072977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.4.207.3:40431)
0923 22:17:37.073165 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.073204 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.074308 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.074572 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.074576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.075039 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:37.075270 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.075276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.077418 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.077418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.078135 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.078137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.078617 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.078618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.078694 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.079110 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:37.079116 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:37.081447 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:37.084538 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:37.084538 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:37.084539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:37.084740 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:37.084982 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:37.084983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:37.084983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:37.085055 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:37.085365 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:37.085366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:37.085427 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.085492 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.085593 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":4,"spinlock_wait_cycles":8960,"thread_start_us":103,"threads_started":1}
I20260923 22:17:37.086048 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.072960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:40431 (local address 127.4.207.61:44157)
0923 22:17:37.073007 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.073009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.073342 (+ 333us) server_negotiation.cc:407] Connection header received
0923 22:17:37.074427 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.074428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.074441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.074493 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.075381 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.076167 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.078222 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.078524 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.078598 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.081723 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.081724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.081725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.081732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.084459 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.084819 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.084821 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.084821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.084883 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.085133 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.085134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.085134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.085277 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.085833 (+ 556us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.085879 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.085960 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:37.086129 11037 ts_manager.cc:194] Re-registered known tserver with Master: 6cc23253566941d4b30895e230e765ae (127.4.207.3:38899)
I20260923 22:17:37.086594 11037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:58123
I20260923 22:17:37.088016 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.072960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:52683 (local address 127.4.207.60:40935)
0923 22:17:37.073407 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.073409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.073470 (+ 61us) server_negotiation.cc:407] Connection header received
0923 22:17:37.074650 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.074652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.074663 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.074722 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.075381 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.076160 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.077727 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.077910 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.077995 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.081277 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.081278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.081278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.081286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.083030 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.083422 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.083424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.083424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.083510 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.083707 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.083708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.083709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.083833 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.083891 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.087816 (+ 3925us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.087925 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
I20260923 22:17:37.088296 12273 heartbeater.cc:347] Connected to a master server at 127.4.207.60:40935
I20260923 22:17:37.088346 12273 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:37.088406 12273 heartbeater.cc:511] Master 127.4.207.60:40935 requested a full tablet report, sending...
I20260923 22:17:37.088521 12275 heartbeater.cc:347] Connected to a master server at 127.4.207.61:44157
I20260923 22:17:37.088572 12275 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:37.088624 12275 heartbeater.cc:511] Master 127.4.207.61:44157 requested a full tablet report, sending...
I20260923 22:17:37.088704 11821 ts_manager.cc:194] Registered new tserver with Master: 6cc23253566941d4b30895e230e765ae (127.4.207.3:38899)
I20260923 22:17:37.088917 11635 ts_manager.cc:194] Registered new tserver with Master: 6cc23253566941d4b30895e230e765ae (127.4.207.3:38899)
I20260923 22:17:37.092751 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.084007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.4.207.62:41877)
0923 22:17:37.084049 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.084051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.085108 (+ 1057us) server_negotiation.cc:407] Connection header received
0923 22:17:37.085153 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.085154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.085167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.085217 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.085971 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.086698 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.087859 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.088134 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.088177 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.089872 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.089874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.089874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.089882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.091472 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.091887 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.091889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.091889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.091948 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.092166 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.092167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.092168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.092312 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.092420 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.092559 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.092634 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:37.093684 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.083790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.4.207.60:40935)
0923 22:17:37.085193 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.085196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.085203 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:37.085231 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.085231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.085241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.085299 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.086158 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.086948 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.087983 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.088913 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.088966 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.090518 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.090520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.090521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.090528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.092357 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.092835 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.092836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.092837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.092898 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.093076 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.093077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.093078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.093220 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.093321 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.093462 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.093568 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1385,"thread_start_us":55,"threads_started":1}
I20260923 22:17:37.094447 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.084443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59372 (local address 127.4.207.61:44157)
0923 22:17:37.086316 (+ 1873us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.086318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.086324 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:17:37.086339 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.086340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.086348 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.086389 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.087051 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.087797 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.089107 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.089374 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.089407 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.091039 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.091040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.091040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.091047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.092685 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.093040 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.093041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.093041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.093111 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.093319 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.093320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.093321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.093443 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.094057 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.094299 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.094387 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1861,"thread_start_us":65,"threads_started":1}
I20260923 22:17:37.094473 11037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:37.096341 11037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:37.104377 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.100115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:41353 (local address 127.0.0.1:41700)
0923 22:17:37.100158 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.100169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.100228 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.100688 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.100689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.100690 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:37.100729 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.100731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.101998 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.101999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.103107 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.103109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.104138 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.104139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.104199 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.104203 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.104254 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.104299 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 22:17:37.104708 12293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.100169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:38899 (local address 127.0.0.1:33378)
0923 22:17:37.100341 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.100350 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.100386 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.101254 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.101255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.101257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:37.101298 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.101300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.102506 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.102507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.103591 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.103594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.104502 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.104503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.104578 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.104582 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.104612 (+ 30us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.104648 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":129,"threads_started":1}
I20260923 22:17:37.105032 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.100319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.4.207.1:41353)
0923 22:17:37.100498 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.100504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.100515 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:17:37.100564 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.100566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.100582 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.100692 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:37.100772 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.101904 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.103198 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.103968 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.104034 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.104105 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.104906 (+ 801us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.104953 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260923 22:17:37.105095 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.100475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.4.207.3:38899)
0923 22:17:37.101033 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.101038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.101047 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:37.101089 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.101091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.101106 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.101365 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:37.101439 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.102397 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.103682 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.104401 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.104864 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.104950 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.104972 (+ 22us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.105011 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":483,"threads_started":1}
I20260923 22:17:37.105720 12207 tablet_service.cc:1467] Processing CreateTablet for tablet 9a94f249f273487dbeb1b10edbfdda64 (DEFAULT_TABLE table=test-table [id=752a0638844d4b08a3a54be5320701f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:37.105731 11934 tablet_service.cc:1467] Processing CreateTablet for tablet 9a94f249f273487dbeb1b10edbfdda64 (DEFAULT_TABLE table=test-table [id=752a0638844d4b08a3a54be5320701f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:37.106077 12207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a94f249f273487dbeb1b10edbfdda64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:37.106076 11934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a94f249f273487dbeb1b10edbfdda64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:37.107831 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.101653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.0.0.1:42484)
0923 22:17:37.101797 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.101808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.101855 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.102049 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.102051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.102053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:37.102098 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.102100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.103244 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.103245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.104431 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.104434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.107588 (+ 3154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.107589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.107640 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.107643 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.107708 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.107749 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260923 22:17:37.107836 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.101808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.4.207.60:40935)
0923 22:17:37.101841 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.101843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.101850 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:37.101909 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.101910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.101946 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.102003 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:37.102182 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.102898 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.104526 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.105141 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.105187 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.105263 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.107717 (+ 2454us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.107766 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:17:37.110116 12299 tablet_bootstrap.cc:492] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: Bootstrap starting.
I20260923 22:17:37.110313 12300 tablet_bootstrap.cc:492] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Bootstrap starting.
I20260923 22:17:37.110365 11831 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260923 22:17:37.110891 12299 tablet_bootstrap.cc:654] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:37.110970 12294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.100295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33925 (local address 127.0.0.1:56112)
0923 22:17:37.100427 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.100438 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.100488 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.102248 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.102249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.102250 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:37.102282 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.102283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.103535 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.103536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.104574 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.104576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.110761 (+ 6185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.110762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.110824 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.110829 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.110858 (+ 29us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.110902 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260923 22:17:37.111255 12299 log.cc:824] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:37.111285 12300 tablet_bootstrap.cc:654] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:37.111285 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88824b9d36574c1a8bae6e1da0e97a81. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:37.111387 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: This master's current role is: LEARNER
I20260923 22:17:37.111611 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.101114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.4.207.2:33925)
0923 22:17:37.102077 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.102082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.102092 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:37.102127 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.102129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.102144 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.102346 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:37.102400 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.103400 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.104660 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.110663 (+ 6003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.111155 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.111256 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.111285 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.111336 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":927,"thread_start_us":84,"threads_started":1}
I20260923 22:17:37.111792 12300 log.cc:824] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:37.112280 12071 tablet_service.cc:1467] Processing CreateTablet for tablet 9a94f249f273487dbeb1b10edbfdda64 (DEFAULT_TABLE table=test-table [id=752a0638844d4b08a3a54be5320701f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:37.112605 12300 tablet_bootstrap.cc:492] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: No bootstrap required, opened a new log
I20260923 22:17:37.112612 12071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a94f249f273487dbeb1b10edbfdda64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:37.112668 12300 ts_tablet_manager.cc:1397] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:37.112702 12299 tablet_bootstrap.cc:492] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: No bootstrap required, opened a new log
I20260923 22:17:37.112756 12299 ts_tablet_manager.cc:1397] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 22:17:37.115404 12306 tablet_bootstrap.cc:492] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Bootstrap starting.
I20260923 22:17:37.115654 12300 raft_consensus.cc:348] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.115654 12299 raft_consensus.cc:348] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.115813 12300 raft_consensus.cc:374] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:37.115813 12299 raft_consensus.cc:374] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:37.115847 12300 raft_consensus.cc:729] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b651870caa74b9ba700c170af10218c, State: Initialized, Role: FOLLOWER
I20260923 22:17:37.115847 12299 raft_consensus.cc:729] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc23253566941d4b30895e230e765ae, State: Initialized, Role: FOLLOWER
I20260923 22:17:37.115967 12300 consensus_queue.cc:260] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.115967 12299 consensus_queue.cc:260] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.116034 12306 tablet_bootstrap.cc:654] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:37.116235 12299 ts_tablet_manager.cc:1428] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260923 22:17:37.116235 12300 ts_tablet_manager.cc:1428] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 22:17:37.116340 12306 log.cc:824] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:37.116802 12274 heartbeater.cc:503] Master 127.4.207.62:41877 was elected leader, sending a full tablet report...
I20260923 22:17:37.116820 12001 heartbeater.cc:503] Master 127.4.207.62:41877 was elected leader, sending a full tablet report...
I20260923 22:17:37.117501 12306 tablet_bootstrap.cc:492] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: No bootstrap required, opened a new log
I20260923 22:17:37.117588 12306 ts_tablet_manager.cc:1397] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:37.119352 12306 raft_consensus.cc:348] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.119537 12306 raft_consensus.cc:374] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:37.119583 12306 raft_consensus.cc:729] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3706851c2474a1cb29162189eeb077d, State: Initialized, Role: FOLLOWER
I20260923 22:17:37.119710 12306 consensus_queue.cc:260] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.119963 12306 ts_tablet_manager.cc:1428] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:17:37.120028 12138 heartbeater.cc:503] Master 127.4.207.62:41877 was elected leader, sending a full tablet report...
W20260923 22:17:37.182054 12140 tablet.cc:2369] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:37.253646 12311 raft_consensus.cc:482] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:37.253758 12311 raft_consensus.cc:504] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.254217 12311 leader_election.cc:288] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353), 6cc23253566941d4b30895e230e765ae (127.4.207.3:38899)
I20260923 22:17:37.258225 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.254503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:34677 (local address 127.4.207.3:38899)
0923 22:17:37.254549 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.254551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.254561 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:37.254593 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.254594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.254603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.254670 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:37.254894 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.255664 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.257186 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.257729 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.257801 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.257811 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.258108 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.258161 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:37.258247 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.254530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:55079 (local address 127.4.207.1:41353)
0923 22:17:37.254575 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.254577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.254584 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:37.254607 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.254608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.254619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.254685 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:37.254898 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.255664 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.257173 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.257730 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.257801 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.257811 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.258108 (+ 297us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.258156 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:37.258173 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.254436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:41353 (local address 127.4.207.2:55079)
0923 22:17:37.254466 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.254474 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.254512 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.254699 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.254700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.254701 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:37.254802 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.254804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.255778 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.255779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.257071 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.257074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.257816 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.257817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.257912 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.257917 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.258014 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.258075 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":92}
I20260923 22:17:37.258553 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.254396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:38899 (local address 127.4.207.2:34677)
0923 22:17:37.254453 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:37.254466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:37.254513 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:37.254699 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:37.254700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:37.254701 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:37.254802 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.254804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.255778 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.255779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.257071 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:37.257074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.257816 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.257817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:37.257912 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.257917 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:37.258009 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 22:17:37.258075 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":85}
I20260923 22:17:37.258729 11954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a94f249f273487dbeb1b10edbfdda64" candidate_uuid: "d3706851c2474a1cb29162189eeb077d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b651870caa74b9ba700c170af10218c" is_pre_election: true
I20260923 22:17:37.258901 11954 raft_consensus.cc:2454] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3706851c2474a1cb29162189eeb077d in term 0.
I20260923 22:17:37.258920 12227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a94f249f273487dbeb1b10edbfdda64" candidate_uuid: "d3706851c2474a1cb29162189eeb077d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc23253566941d4b30895e230e765ae" is_pre_election: true
I20260923 22:17:37.259097 12227 raft_consensus.cc:2454] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3706851c2474a1cb29162189eeb077d in term 0.
I20260923 22:17:37.259135 12024 leader_election.cc:302] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b651870caa74b9ba700c170af10218c, d3706851c2474a1cb29162189eeb077d; no voters:
I20260923 22:17:37.259315 12311 raft_consensus.cc:2781] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:37.259392 12311 raft_consensus.cc:482] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:37.259424 12311 raft_consensus.cc:3037] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:37.260212 12311 raft_consensus.cc:504] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.260370 12311 leader_election.cc:288] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [CANDIDATE]: Term 1 election: Requested vote from peers 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353), 6cc23253566941d4b30895e230e765ae (127.4.207.3:38899)
I20260923 22:17:37.260515 12227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a94f249f273487dbeb1b10edbfdda64" candidate_uuid: "d3706851c2474a1cb29162189eeb077d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc23253566941d4b30895e230e765ae"
I20260923 22:17:37.260546 11954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a94f249f273487dbeb1b10edbfdda64" candidate_uuid: "d3706851c2474a1cb29162189eeb077d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b651870caa74b9ba700c170af10218c"
I20260923 22:17:37.260597 12227 raft_consensus.cc:3037] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:37.260607 11954 raft_consensus.cc:3037] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:37.261477 12227 raft_consensus.cc:2454] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3706851c2474a1cb29162189eeb077d in term 1.
I20260923 22:17:37.261477 11954 raft_consensus.cc:2454] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3706851c2474a1cb29162189eeb077d in term 1.
I20260923 22:17:37.261685 12026 leader_election.cc:302] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc23253566941d4b30895e230e765ae, d3706851c2474a1cb29162189eeb077d; no voters:
I20260923 22:17:37.261783 12311 raft_consensus.cc:2781] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:37.261942 12311 raft_consensus.cc:686] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 1 LEADER]: Becoming Leader. State: Replica: d3706851c2474a1cb29162189eeb077d, State: Running, Role: LEADER
I20260923 22:17:37.262156 12311 consensus_queue.cc:237] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:37.263123 11036 catalog_manager.cc:5495] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d reported cstate change: term changed from 0 to 1, leader changed from <none> to d3706851c2474a1cb29162189eeb077d (127.4.207.2). New cstate: current_term: 1 leader_uuid: "d3706851c2474a1cb29162189eeb077d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } health_report { overall_health: HEALTHY } } }
I20260923 22:17:37.263804 12314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: NON_VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: true } } }
I20260923 22:17:37.263900 12314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:37.264518 11596 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: attempting to promote NON_VOTER 8f6d32aa7811463cbd1591740ebf9638 to VOTER
I20260923 22:17:37.264653 11596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } }
I20260923 22:17:37.265128 11645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Refusing update from remote peer 88824b9d36574c1a8bae6e1da0e97a81: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260923 22:17:37.265177 11831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 LEARNER]: Refusing update from remote peer 88824b9d36574c1a8bae6e1da0e97a81: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260923 22:17:37.265367 12316 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260923 22:17:37.265452 12314 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260923 22:17:37.266023 12314 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8f6d32aa7811463cbd1591740ebf9638 (127.4.207.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:37.266322 11645 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8f6d32aa7811463cbd1591740ebf9638 (127.4.207.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:37.266291 11831 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8f6d32aa7811463cbd1591740ebf9638 (127.4.207.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:37.266825 12314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:37.266914 12314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:37.266885 11596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:37.266940 11596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:37.266965 12314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:37.267030 12314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:37.267253 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:37.267345 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:37.267514 12317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "88824b9d36574c1a8bae6e1da0e97a81" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:37.267607 12317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:37.287983 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.285529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.4.207.2:33925)
0923 22:17:37.285594 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.285596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.285608 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:37.285643 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.285644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.285663 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.285746 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:37.285902 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.286396 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.287166 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.287359 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.287469 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.287646 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:37.287750 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:37.287857 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.287876 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.287896 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:17:37.289842 11954 raft_consensus.cc:1261] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 1 FOLLOWER]: Refusing update from remote peer d3706851c2474a1cb29162189eeb077d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:37.289858 12227 raft_consensus.cc:1261] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 1 FOLLOWER]: Refusing update from remote peer d3706851c2474a1cb29162189eeb077d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:37.290174 12311 consensus_queue.cc:1035] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:37.290338 12312 consensus_queue.cc:1035] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:17:37.290427 12003 tablet.cc:2369] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:37.293133 12319 mvcc.cc:205] Tried to move back new op lower bound from 7332666807456395264 to 7332666807346388992. Current Snapshot: MvccSnapshot[applied={T|T < 7332666807456395264}]
I20260923 22:17:37.293351 12322 mvcc.cc:205] Tried to move back new op lower bound from 7332666807456395264 to 7332666807346388992. Current Snapshot: MvccSnapshot[applied={T|T < 7332666807456395264}]
I20260923 22:17:37.295990 12320 mvcc.cc:205] Tried to move back new op lower bound from 7332666807456395264 to 7332666807346388992. Current Snapshot: MvccSnapshot[applied={T|T < 7332666807456395264 or (T in {7332666807456395264})}]
I20260923 22:17:37.296980 11036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:37.297075 11036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:37.298467 11036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88824b9d36574c1a8bae6e1da0e97a81: Sending DeleteTablet for 3 replicas of tablet 9a94f249f273487dbeb1b10edbfdda64
I20260923 22:17:37.298820 12207 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a94f249f273487dbeb1b10edbfdda64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:37 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:33378
I20260923 22:17:37.298924 11934 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a94f249f273487dbeb1b10edbfdda64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:37 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:41700
I20260923 22:17:37.299072 12329 tablet_replica.cc:330] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: stopping tablet replica
I20260923 22:17:37.299188 12329 raft_consensus.cc:2229] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:37.299276 12330 tablet_replica.cc:330] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: stopping tablet replica
I20260923 22:17:37.299340 12329 raft_consensus.cc:2258] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:37.299850 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 11005
I20260923 22:17:37.300246 12071 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a94f249f273487dbeb1b10edbfdda64 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:37 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:56112
I20260923 22:17:37.301070 12329 ts_tablet_manager.cc:1905] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:37.299487 12330 raft_consensus.cc:2229] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:37.301318 12330 raft_consensus.cc:2258] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:37.303373 12329 ts_tablet_manager.cc:1918] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:37.303484 12329 log.cc:1196] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/9a94f249f273487dbeb1b10edbfdda64
I20260923 22:17:37.303593 12330 ts_tablet_manager.cc:1905] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:37.303771 12329 ts_tablet_manager.cc:1939] T 9a94f249f273487dbeb1b10edbfdda64 P 6cc23253566941d4b30895e230e765ae: Deleting consensus metadata
I20260923 22:17:37.305488 12330 ts_tablet_manager.cc:1918] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:37.305627 12330 log.cc:1196] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/9a94f249f273487dbeb1b10edbfdda64
W20260923 22:17:37.305896 12162 connection.cc:535] server connection from 127.0.0.1:33378 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 22:17:37.306269 12330 ts_tablet_manager.cc:1939] T 9a94f249f273487dbeb1b10edbfdda64 P 8b651870caa74b9ba700c170af10218c: Deleting consensus metadata
I20260923 22:17:37.306545 12332 tablet_replica.cc:330] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: stopping tablet replica
I20260923 22:17:37.306676 12332 raft_consensus.cc:2229] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:37.306788 12332 raft_consensus.cc:2258] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 22:17:37.306798 11889 connection.cc:441] server connection from 127.0.0.1:41700 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41700 (request call id 1) could send its response
I20260923 22:17:37.307301 12332 ts_tablet_manager.cc:1905] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:37.309214 12332 ts_tablet_manager.cc:1918] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:37.309293 12332 log.cc:1196] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/9a94f249f273487dbeb1b10edbfdda64
I20260923 22:17:37.309546 12332 ts_tablet_manager.cc:1939] T 9a94f249f273487dbeb1b10edbfdda64 P d3706851c2474a1cb29162189eeb077d: Deleting consensus metadata
W20260923 22:17:37.309880 12027 connection.cc:441] server connection from 127.0.0.1:56112 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56112 (request call id 1) could send its response
I20260923 22:17:37.316968 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.307656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.4.207.61:44157)
0923 22:17:37.307702 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.307704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.308041 (+ 337us) server_negotiation.cc:407] Connection header received
0923 22:17:37.308144 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.308146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.308157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.308187 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.309024 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.309779 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.310766 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.311045 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.311091 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.312986 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.312988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.312988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.312996 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.315654 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.316074 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.316075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.316075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.316144 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.316375 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.316375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.316376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.316511 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.316572 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.316777 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.316863 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:37.317099 12285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.299682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.4.207.60:40935)
0923 22:17:37.299725 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.299727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.307063 (+ 7336us) server_negotiation.cc:407] Connection header received
0923 22:17:37.307180 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.307181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.307193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.307225 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.308190 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.308984 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.310311 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.310620 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.310694 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.312987 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.312988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.312988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.312996 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.314690 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.315190 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.315191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.315192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.315249 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.315828 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.315830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.315830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.315972 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.316365 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.316557 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.317003 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:37.318689 12004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:37.310474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.4.207.60:40935)
0923 22:17:37.310517 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:37.310518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:37.310670 (+ 152us) server_negotiation.cc:407] Connection header received
0923 22:17:37.310762 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:37.310763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:37.310772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:37.310800 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:37.311535 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.312291 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.313297 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:37.313598 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:37.313645 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:37.315155 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:37.315156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:37.315156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:37.315163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:37.316924 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.318076 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.318077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.318082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.318129 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:37.318314 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:37.318314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:37.318314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:37.318393 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:37.318455 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:37.318579 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 22:17:37.318634 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:38.295529 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.295239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.3:60477)
0923 22:17:38.295356 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.295429 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
W20260923 22:17:38.295883 12274 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:41877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
I20260923 22:17:38.295935 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.295641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.2:57549)
0923 22:17:38.295829 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.295892 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":66,"threads_started":1}
I20260923 22:17:38.296000 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.295753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.1:52913)
0923 22:17:38.295881 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.295949 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
W20260923 22:17:38.296144 12138 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:41877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
W20260923 22:17:38.296350 12001 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:41877 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
I20260923 22:17:38.798806 12342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88824b9d36574c1a8bae6e1da0e97a81)
I20260923 22:17:38.798939 12343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88824b9d36574c1a8bae6e1da0e97a81)
I20260923 22:17:38.799005 12342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } }
I20260923 22:17:38.799103 12343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } }
I20260923 22:17:38.799368 12342 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88824b9d36574c1a8bae6e1da0e97a81 (127.4.207.62:41877), 7d129f5f04484696b5aec295fb9f1b4b (127.4.207.61:44157)
I20260923 22:17:38.799595 12343 leader_election.cc:288] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88824b9d36574c1a8bae6e1da0e97a81 (127.4.207.62:41877), 8f6d32aa7811463cbd1591740ebf9638 (127.4.207.60:40935)
I20260923 22:17:38.799754 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.799506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43568)
0923 22:17:38.799618 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.799679 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
I20260923 22:17:38.799816 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.799530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43566)
0923 22:17:38.799690 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.799774 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":58,"threads_started":1}
W20260923 22:17:38.799853 11622 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
W20260923 22:17:38.799970 11808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
I20260923 22:17:38.800453 12344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.800344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43584)
0923 22:17:38.800396 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.800423 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 22:17:38.800530 11622 leader_election.cc:334] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88824b9d36574c1a8bae6e1da0e97a81 (127.4.207.62:41877): Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
I20260923 22:17:38.800547 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.800448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43598)
0923 22:17:38.800497 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.800519 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 22:17:38.800621 11808 leader_election.cc:334] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88824b9d36574c1a8bae6e1da0e97a81 (127.4.207.62:41877): Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
I20260923 22:17:38.803023 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.799695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:44157 (local address 127.0.0.1:59384)
0923 22:17:38.799785 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.799797 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:38.799896 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:38.800157 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:38.800158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:38.800161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:38.800235 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.800238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.801214 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.801215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.802003 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.802005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.802746 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.802747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.802832 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.802836 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:38.802911 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 22:17:38.802957 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":7,"thread_start_us":57,"threads_started":1}
I20260923 22:17:38.803018 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.799842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.4.207.61:44157)
0923 22:17:38.799984 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:38.799986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:38.799994 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:38.800021 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:38.800022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:38.800059 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:38.800301 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:38.800335 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.801143 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.802100 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.802650 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.802743 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.802756 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:38.802919 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 22:17:38.802958 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":52,"threads_started":1}
I20260923 22:17:38.803232 11645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f6d32aa7811463cbd1591740ebf9638" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7d129f5f04484696b5aec295fb9f1b4b" is_pre_election: true
I20260923 22:17:38.803261 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.799664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.4.207.60:40935)
0923 22:17:38.799995 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:38.799997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:38.800166 (+ 169us) server_negotiation.cc:407] Connection header received
0923 22:17:38.800185 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:38.800185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:38.800194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:38.800253 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:38.800438 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.801315 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.802181 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.802901 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.802992 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.803001 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:38.803155 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 22:17:38.803203 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":315,"thread_start_us":53,"threads_started":1}
I20260923 22:17:38.803339 11645 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f6d32aa7811463cbd1591740ebf9638 in term 1.
I20260923 22:17:38.803488 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.799526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:40935 (local address 127.0.0.1:42512)
0923 22:17:38.800052 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.800062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:38.800113 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:38.800281 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:38.800282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:38.800285 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:38.800341 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.800343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.801387 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.801388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.802089 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.802091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.802994 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.802995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.803061 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.803066 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:38.803358 (+ 292us) client_negotiation.cc:240] Negotiation successful
0923 22:17:38.803401 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":38,"threads_started":1}
I20260923 22:17:38.803503 11808 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d129f5f04484696b5aec295fb9f1b4b, 8f6d32aa7811463cbd1591740ebf9638; no voters: 88824b9d36574c1a8bae6e1da0e97a81
I20260923 22:17:38.803624 12342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 22:17:38.803661 12342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88824b9d36574c1a8bae6e1da0e97a81)
I20260923 22:17:38.803692 12342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:17:38.803720 11831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7d129f5f04484696b5aec295fb9f1b4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8f6d32aa7811463cbd1591740ebf9638" is_pre_election: true
I20260923 22:17:38.804415 12342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } }
I20260923 22:17:38.804579 12342 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [CANDIDATE]: Term 2 election: Requested vote from peers 88824b9d36574c1a8bae6e1da0e97a81 (127.4.207.62:41877), 7d129f5f04484696b5aec295fb9f1b4b (127.4.207.61:44157)
I20260923 22:17:38.804579 11831 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d129f5f04484696b5aec295fb9f1b4b in current term 2: Already voted for candidate 8f6d32aa7811463cbd1591740ebf9638 in this term.
I20260923 22:17:38.804780 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.804670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43614)
0923 22:17:38.804739 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.804760 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 22:17:38.804843 11619 leader_election.cc:302] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d129f5f04484696b5aec295fb9f1b4b; no voters: 88824b9d36574c1a8bae6e1da0e97a81, 8f6d32aa7811463cbd1591740ebf9638
I20260923 22:17:38.804951 12343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:17:38.805073 11645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f6d32aa7811463cbd1591740ebf9638" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7d129f5f04484696b5aec295fb9f1b4b"
I20260923 22:17:38.805229 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.805107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43622)
0923 22:17:38.805182 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.805209 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260923 22:17:38.805344 11808 leader_election.cc:334] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 88824b9d36574c1a8bae6e1da0e97a81 (127.4.207.62:41877): Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
I20260923 22:17:38.805752 12343 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:17:38.806545 11645 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f6d32aa7811463cbd1591740ebf9638 in term 2.
I20260923 22:17:38.806702 11808 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7d129f5f04484696b5aec295fb9f1b4b, 8f6d32aa7811463cbd1591740ebf9638; no voters: 88824b9d36574c1a8bae6e1da0e97a81
I20260923 22:17:38.806846 12342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:17:38.807000 12342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [term 2 LEADER]: Becoming Leader. State: Replica: 8f6d32aa7811463cbd1591740ebf9638, State: Running, Role: LEADER
I20260923 22:17:38.807096 12342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } }
I20260923 22:17:38.807452 12352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6d32aa7811463cbd1591740ebf9638. Latest consensus state: current_term: 2 leader_uuid: "8f6d32aa7811463cbd1591740ebf9638" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:38.807548 12352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:38.807727 12354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:38.808146 12354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:38.808241 12354 catalog_manager.cc:1259] Loaded cluster ID: 9639e6723cfd4048a92e3666e08b6bf7
I20260923 22:17:38.808274 12354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:38.808445 12354 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:38.808543 12354 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Loaded TSK: 0
I20260923 22:17:38.808681 12354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:17:38.859838 11820 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:38.861481 11820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:38.862617 11645 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [term 2 FOLLOWER]: Refusing update from remote peer 8f6d32aa7811463cbd1591740ebf9638: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 22:17:38.862798 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.862533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43632)
0923 22:17:38.862675 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.862711 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260923 22:17:38.862960 12352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 22:17:38.863220 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.863092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43646)
0923 22:17:38.863167 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.863203 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 22:17:38.863582 11808 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 -> Peer 88824b9d36574c1a8bae6e1da0e97a81 (127.4.207.62:41877): Couldn't send request to peer 88824b9d36574c1a8bae6e1da0e97a81. Status: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 22:17:38.863586 12343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6d32aa7811463cbd1591740ebf9638. Latest consensus state: current_term: 2 leader_uuid: "8f6d32aa7811463cbd1591740ebf9638" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:38.863711 12343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:38.863952 12355 mvcc.cc:205] Tried to move back new op lower bound from 7332666813899026432 to 7332666813674500096. Current Snapshot: MvccSnapshot[applied={T|T < 7332666813899026432}]
I20260923 22:17:38.864053 12342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8f6d32aa7811463cbd1591740ebf9638" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:38.864156 12342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638 [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:38.864214 12343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8f6d32aa7811463cbd1591740ebf9638" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "88824b9d36574c1a8bae6e1da0e97a81" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41877 } } peers { permanent_uuid: "7d129f5f04484696b5aec295fb9f1b4b" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 44157 } attrs { promote: false } } peers { permanent_uuid: "8f6d32aa7811463cbd1591740ebf9638" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 40935 } attrs { promote: false } } }
I20260923 22:17:38.864308 12343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d129f5f04484696b5aec295fb9f1b4b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:17:38.869318 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.866022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.4.207.1:41353)
0923 22:17:38.866366 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:38.866369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:38.866377 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:38.866403 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:38.866403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:38.866414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:38.866503 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:38.866623 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.867122 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.868420 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.868993 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.869082 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.869096 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:38.869207 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 22:17:38.869245 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":50,"threads_started":1}
I20260923 22:17:38.869310 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.865913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:41353 (local address 127.0.0.1:41710)
0923 22:17:38.865951 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.865960 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:38.865998 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:38.866510 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:38.866511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:38.866512 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:38.866562 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.866563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.867204 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.867205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.868322 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.868325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.869081 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.869082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.869140 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.869145 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:38.869198 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:17:38.869238 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1}
I20260923 22:17:38.869570 11934 tablet_service.cc:1467] Processing CreateTablet for tablet ea2b9b95d67741f39c23511af564f67b (DEFAULT_TABLE table=test-table [id=2b84caf6d20c458499ceb49005c4c2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:38.869668 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.865959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:38899 (local address 127.0.0.1:33392)
0923 22:17:38.866086 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.866094 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:38.866165 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:38.866500 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:38.866502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:38.866504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:38.866564 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.866565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.867507 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.867508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.868667 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.868669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.869407 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.869408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.869485 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.869491 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:17:38.869533 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 22:17:38.869575 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":23}
I20260923 22:17:38.869692 11934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2b9b95d67741f39c23511af564f67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:38.869817 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.865975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33925 (local address 127.0.0.1:56136)
0923 22:17:38.866085 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.866095 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:38.866142 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:38.866734 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:38.866736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:38.866738 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:38.866768 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.866769 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.867668 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.867669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.868583 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:38.868585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.869630 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.869630 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:38.869685 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.869689 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:38.869724 (+ 35us) client_negotiation.cc:240] Negotiation successful
0923 22:17:38.869763 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":77,"threads_started":1}
I20260923 22:17:38.869900 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.866192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.4.207.3:38899)
0923 22:17:38.866325 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:38.866328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:38.866337 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:38.866364 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:38.866365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:38.866377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:38.866447 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:38.866661 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.867419 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.868730 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.869293 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.869359 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.869373 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:38.869759 (+ 386us) server_negotiation.cc:299] Negotiation successful
0923 22:17:38.869816 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260923 22:17:38.870250 12207 tablet_service.cc:1467] Processing CreateTablet for tablet ea2b9b95d67741f39c23511af564f67b (DEFAULT_TABLE table=test-table [id=2b84caf6d20c458499ceb49005c4c2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:38.870363 12207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2b9b95d67741f39c23511af564f67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:38.870891 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.866229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.4.207.2:33925)
0923 22:17:38.866339 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:38.866341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:38.866348 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:38.866370 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:38.866370 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:38.866378 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:38.866829 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:38.866868 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.867589 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.868652 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:38.869289 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:38.870456 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:38.870474 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:38.870495 (+ 21us) server_negotiation.cc:299] Negotiation successful
0923 22:17:38.870540 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":74,"threads_started":1}
I20260923 22:17:38.871009 12365 tablet_bootstrap.cc:492] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c: Bootstrap starting.
I20260923 22:17:38.871290 12071 tablet_service.cc:1467] Processing CreateTablet for tablet ea2b9b95d67741f39c23511af564f67b (DEFAULT_TABLE table=test-table [id=2b84caf6d20c458499ceb49005c4c2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:38.871407 12071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea2b9b95d67741f39c23511af564f67b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:38.871403 12365 tablet_bootstrap.cc:654] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:38.871982 12365 tablet_bootstrap.cc:492] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c: No bootstrap required, opened a new log
I20260923 22:17:38.872058 12365 ts_tablet_manager.cc:1397] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:38.872206 12365 raft_consensus.cc:348] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:38.872288 12365 raft_consensus.cc:374] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:38.872321 12365 raft_consensus.cc:729] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b651870caa74b9ba700c170af10218c, State: Initialized, Role: FOLLOWER
I20260923 22:17:38.872372 12365 consensus_queue.cc:260] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:38.872550 12365 ts_tablet_manager.cc:1428] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:38.872943 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.872706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.1:59573)
0923 22:17:38.872862 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.872908 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260923 22:17:38.873050 12369 tablet_bootstrap.cc:492] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: Bootstrap starting.
I20260923 22:17:38.873075 12367 tablet_bootstrap.cc:492] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: Bootstrap starting.
I20260923 22:17:38.873099 12000 heartbeater.cc:503] Master 127.4.207.60:40935 was elected leader, sending a full tablet report...
I20260923 22:17:38.873472 12367 tablet_bootstrap.cc:654] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:38.873486 12369 tablet_bootstrap.cc:654] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:38.874007 12367 tablet_bootstrap.cc:492] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: No bootstrap required, opened a new log
I20260923 22:17:38.874032 12369 tablet_bootstrap.cc:492] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: No bootstrap required, opened a new log
I20260923 22:17:38.874025 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.873914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.1:52839)
0923 22:17:38.873977 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.874000 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 22:17:38.874082 12369 ts_tablet_manager.cc:1397] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:38.874085 12367 ts_tablet_manager.cc:1397] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:38.874219 12369 raft_consensus.cc:348] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:38.874286 12369 raft_consensus.cc:374] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:38.874224 12367 raft_consensus.cc:348] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:38.874311 12367 raft_consensus.cc:374] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:38.874310 12369 raft_consensus.cc:729] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3706851c2474a1cb29162189eeb077d, State: Initialized, Role: FOLLOWER
I20260923 22:17:38.874346 12367 raft_consensus.cc:729] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc23253566941d4b30895e230e765ae, State: Initialized, Role: FOLLOWER
I20260923 22:17:38.874387 12369 consensus_queue.cc:260] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:38.874396 12367 consensus_queue.cc:260] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:38.874517 12369 ts_tablet_manager.cc:1428] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:38.874524 12367 ts_tablet_manager.cc:1428] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:38.875017 12273 heartbeater.cc:503] Master 127.4.207.60:40935 was elected leader, sending a full tablet report...
I20260923 22:17:38.875064 12137 heartbeater.cc:503] Master 127.4.207.60:40935 was elected leader, sending a full tablet report...
I20260923 22:17:38.875288 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.874822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.3:52219)
0923 22:17:38.875212 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.875255 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":73,"threads_started":1}
I20260923 22:17:38.875558 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.874689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.2:47199)
0923 22:17:38.875494 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.875524 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":73,"threads_started":1}
I20260923 22:17:38.875748 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:38.875663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.3:38033)
0923 22:17:38.875702 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:38.875730 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 22:17:39.074007 12276 tablet.cc:2369] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:39.217396 12374 raft_consensus.cc:482] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:39.217499 12374 raft_consensus.cc:504] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:39.217921 12374 leader_election.cc:288] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353), d3706851c2474a1cb29162189eeb077d (127.4.207.2:33925)
I20260923 22:17:39.220904 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.218147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:56891 (local address 127.4.207.1:41353)
0923 22:17:39.218194 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:39.218196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:39.218392 (+ 196us) server_negotiation.cc:407] Connection header received
0923 22:17:39.218425 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:39.218426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:39.218439 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:39.218497 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:39.218663 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.219152 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.220150 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.220528 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.220583 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:39.220593 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:39.220775 (+ 182us) server_negotiation.cc:299] Negotiation successful
0923 22:17:39.220804 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:39.220911 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.218082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:41353 (local address 127.4.207.3:56891)
0923 22:17:39.218312 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.218324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:39.218382 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:39.218516 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:39.218517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:39.218519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:39.218580 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:39.218582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.219239 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.219240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:39.220059 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:39.220061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.220623 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.220624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:39.220689 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:39.220693 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:39.220769 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 22:17:39.220807 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":29,"thread_start_us":129,"threads_started":1}
I20260923 22:17:39.221230 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.218198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:45041 (local address 127.4.207.2:33925)
0923 22:17:39.218244 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:39.218246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:39.218371 (+ 125us) server_negotiation.cc:407] Connection header received
0923 22:17:39.218472 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:39.218490 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:39.218503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:39.218558 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:39.218719 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.219180 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.220326 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.220803 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.220867 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:39.220877 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:39.221114 (+ 237us) server_negotiation.cc:299] Negotiation successful
0923 22:17:39.221184 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:39.221177 11954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2b9b95d67741f39c23511af564f67b" candidate_uuid: "6cc23253566941d4b30895e230e765ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b651870caa74b9ba700c170af10218c" is_pre_election: true
I20260923 22:17:39.221300 11954 raft_consensus.cc:2454] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc23253566941d4b30895e230e765ae in term 0.
I20260923 22:17:39.221462 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.218079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33925 (local address 127.4.207.3:45041)
0923 22:17:39.218285 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.218324 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:39.218385 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:39.218580 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:39.218580 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:39.218582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:39.218645 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:39.218647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.219290 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.219291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:39.220084 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:39.220086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.220890 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.220891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:39.220956 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:39.220961 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:39.221335 (+ 374us) client_negotiation.cc:240] Negotiation successful
0923 22:17:39.221386 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":49}
I20260923 22:17:39.221534 12160 leader_election.cc:302] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc23253566941d4b30895e230e765ae, 8b651870caa74b9ba700c170af10218c; no voters:
I20260923 22:17:39.221670 12374 raft_consensus.cc:2781] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:39.221712 12374 raft_consensus.cc:482] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:39.221740 12374 raft_consensus.cc:3037] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:39.221716 12091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2b9b95d67741f39c23511af564f67b" candidate_uuid: "6cc23253566941d4b30895e230e765ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3706851c2474a1cb29162189eeb077d" is_pre_election: true
I20260923 22:17:39.221868 12091 raft_consensus.cc:2454] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc23253566941d4b30895e230e765ae in term 0.
I20260923 22:17:39.222558 12374 raft_consensus.cc:504] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:39.222711 12374 leader_election.cc:288] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [CANDIDATE]: Term 1 election: Requested vote from peers 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353), d3706851c2474a1cb29162189eeb077d (127.4.207.2:33925)
I20260923 22:17:39.222858 11954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2b9b95d67741f39c23511af564f67b" candidate_uuid: "6cc23253566941d4b30895e230e765ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b651870caa74b9ba700c170af10218c"
I20260923 22:17:39.222936 11954 raft_consensus.cc:3037] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:39.223585 11954 raft_consensus.cc:2454] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc23253566941d4b30895e230e765ae in term 1.
I20260923 22:17:39.223701 12091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea2b9b95d67741f39c23511af564f67b" candidate_uuid: "6cc23253566941d4b30895e230e765ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3706851c2474a1cb29162189eeb077d"
I20260923 22:17:39.223753 12160 leader_election.cc:302] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc23253566941d4b30895e230e765ae, 8b651870caa74b9ba700c170af10218c; no voters:
I20260923 22:17:39.223796 12091 raft_consensus.cc:3037] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:39.223850 12374 raft_consensus.cc:2781] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:39.224066 12374 raft_consensus.cc:686] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 1 LEADER]: Becoming Leader. State: Replica: 6cc23253566941d4b30895e230e765ae, State: Running, Role: LEADER
I20260923 22:17:39.224143 12374 consensus_queue.cc:237] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } }
I20260923 22:17:39.224412 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.224315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.3:45663)
0923 22:17:39.224350 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.224386 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260923 22:17:39.224712 12091 raft_consensus.cc:2454] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc23253566941d4b30895e230e765ae in term 1.
I20260923 22:17:39.224938 11821 catalog_manager.cc:5495] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc23253566941d4b30895e230e765ae (127.4.207.3). New cstate: current_term: 1 leader_uuid: "6cc23253566941d4b30895e230e765ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cc23253566941d4b30895e230e765ae" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 38899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:39.264822 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.264643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43648)
0923 22:17:39.264731 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.264781 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 22:17:39.265200 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.265122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.0.0.1:43662)
0923 22:17:39.265159 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.265183 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 22:17:39.310309 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.308034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.4.207.3:38899)
0923 22:17:39.308094 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:39.308096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:39.308105 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:39.308158 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:39.308159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:39.308183 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:39.308246 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:39.308363 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.308830 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.309564 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:39.309753 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:39.309811 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:39.309965 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:39.310068 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:39.310123 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:39.310165 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 22:17:39.310185 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:39.311556 11954 raft_consensus.cc:1261] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 1 FOLLOWER]: Refusing update from remote peer 6cc23253566941d4b30895e230e765ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:39.311619 12091 raft_consensus.cc:1261] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 1 FOLLOWER]: Refusing update from remote peer 6cc23253566941d4b30895e230e765ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:39.311841 12378 consensus_queue.cc:1035] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b651870caa74b9ba700c170af10218c" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 41353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:39.311993 12374 consensus_queue.cc:1035] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3706851c2474a1cb29162189eeb077d" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:39.312865 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.312751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.2:55183)
0923 22:17:39.312797 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.312830 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 22:17:39.313127 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.312956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.1:50003)
0923 22:17:39.313065 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.313095 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 22:17:39.313572 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.313455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.3:52491)
0923 22:17:39.313518 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.313544 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 22:17:39.314424 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.314316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.3:56613)
0923 22:17:39.314345 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.314373 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260923 22:17:39.314962 12370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.314815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.1:51227)
0923 22:17:39.314892 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.314916 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 22:17:39.314985 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.314898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41877 (local address 127.4.207.2:58783)
0923 22:17:39.314935 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.314955 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:41877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 22:17:39.316903 11821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:39.317004 11821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:39.318306 11821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f6d32aa7811463cbd1591740ebf9638: Sending DeleteTablet for 3 replicas of tablet ea2b9b95d67741f39c23511af564f67b
I20260923 22:17:39.318611 12071 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2b9b95d67741f39c23511af564f67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:39 UTC) from {username='oryx', principal='oryx/127.4.207.60@KRBTEST.COM'} at 127.0.0.1:56136
I20260923 22:17:39.318604 12207 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2b9b95d67741f39c23511af564f67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:39 UTC) from {username='oryx', principal='oryx/127.4.207.60@KRBTEST.COM'} at 127.0.0.1:33392
I20260923 22:17:39.318679 11934 tablet_service.cc:1514] Processing DeleteTablet for tablet ea2b9b95d67741f39c23511af564f67b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:39 UTC) from {username='oryx', principal='oryx/127.4.207.60@KRBTEST.COM'} at 127.0.0.1:41710
I20260923 22:17:39.318782 12390 tablet_replica.cc:330] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: stopping tablet replica
I20260923 22:17:39.318841 12390 raft_consensus.cc:2229] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:39.318921 12390 raft_consensus.cc:2258] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:39.318944 12392 tablet_replica.cc:330] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c: stopping tablet replica
I20260923 22:17:39.319083 12392 raft_consensus.cc:2229] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:39.319130 12391 tablet_replica.cc:330] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: stopping tablet replica
I20260923 22:17:39.319204 12391 raft_consensus.cc:2229] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:39.319217 12392 raft_consensus.cc:2258] T ea2b9b95d67741f39c23511af564f67b P 8b651870caa74b9ba700c170af10218c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:39.319263 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 11783
I20260923 22:17:39.319294 12391 raft_consensus.cc:2258] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:39.320036 12391 ts_tablet_manager.cc:1905] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:39.321543 12391 ts_tablet_manager.cc:1918] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:39.321615 12391 log.cc:1196] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/ea2b9b95d67741f39c23511af564f67b
I20260923 22:17:39.321846 12391 ts_tablet_manager.cc:1939] T ea2b9b95d67741f39c23511af564f67b P 6cc23253566941d4b30895e230e765ae: Deleting consensus metadata
I20260923 22:17:39.322214 11807 catalog_manager.cc:4841] TS 6cc23253566941d4b30895e230e765ae (127.4.207.3:38899): tablet ea2b9b95d67741f39c23511af564f67b (table test-table [id=2b84caf6d20c458499ceb49005c4c2bf]) successfully deleted
I20260923 22:17:39.323704 12390 ts_tablet_manager.cc:1905] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:39.325192 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:39.325037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:41353 (local address 127.0.0.1:41718)
0923 22:17:39.325078 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.325105 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:41353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260923 22:17:39.325297 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 12008
W20260923 22:17:39.325423 11805 catalog_manager.cc:4584] TS 8b651870caa74b9ba700c170af10218c (127.4.207.1:41353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea2b9b95d67741f39c23511af564f67b: Network error: Client connection negotiation failed: client connection to 127.4.207.1:41353: connect: Connection refused (error 111)
I20260923 22:17:39.325558 12390 ts_tablet_manager.cc:1918] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:39.325615 12390 log.cc:1196] T ea2b9b95d67741f39c23511af564f67b P d3706851c2474a1cb29162189eeb077d: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/ea2b9b95d67741f39c23511af564f67b
W20260923 22:17:39.332355 12346 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:39.332197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33925 (local address 127.0.0.1:56140)
0923 22:17:39.332267 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:39.332301 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.2:33925: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 22:17:39.332644 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 12144
I20260923 22:17:39.337651 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 11501
I20260923 22:17:39.342732 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 11689
2026-09-23T22:17:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5071 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:39 dist-test-slave-7511 krb5kdc[12395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:39 dist-test-slave-7511 krb5kdc[12395](info): set up 2 sockets
Sep 23 22:17:39 dist-test-slave-7511 krb5kdc[12395](info): commencing operation
krb5kdc: starting...
W20260923 22:17:40.384083 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:40 dist-test-slave-7511 krb5kdc[12395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.62.keytab.
Entry for principal oryx/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.62.keytab.
I20260923 22:17:40.505854 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:38819
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.62.keytab
--principal=oryx/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:38819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:40.586560 12408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.586812 12408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.586834 12408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.588075 12408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:40.588121 12408 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 22:17:40.588135 12408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.588151 12408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.588205 12408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:40.588230 12408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:40.589794 12408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:38819
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.62.keytab
--principal=oryx/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:38819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:40.590050 12408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.590301 12408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:40.595284 12413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.595312 12414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.595292 12416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.595325 12408 server_base.cc:1034] running on GCE node
I20260923 22:17:40.595810 12408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.596096 12408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.597256 12408 hybrid_clock.cc:648] HybridClock initialized: now 1790201860597246 us; error 36 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-7511 krb5kdc[12395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.599723 12408 init.cc:377] Logged in from keytab as oryx/127.4.207.62@KRBTEST.COM (short username oryx)
I20260923 22:17:40.600172 12408 webserver.cc:460] Webserver started at http://127.4.207.62:38033/ using document root <none> and password file <none>
I20260923 22:17:40.600443 12408 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.600507 12408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.600658 12408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:40.601711 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "36bd61e55bd44451af58188af50b666f"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "093cb6ad3e3635b34936f60d5afcaa87"
server_key_iv: "f4b8769ce9fcee791e56afbb4ec55e13"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.602077 12408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "36bd61e55bd44451af58188af50b666f"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "093cb6ad3e3635b34936f60d5afcaa87"
server_key_iv: "f4b8769ce9fcee791e56afbb4ec55e13"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.603395 12408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:40.604199 12423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.604364 12408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:40.604460 12408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "36bd61e55bd44451af58188af50b666f"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "093cb6ad3e3635b34936f60d5afcaa87"
server_key_iv: "f4b8769ce9fcee791e56afbb4ec55e13"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.604540 12408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.616950 12408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.618777 12408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.618933 12408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.623973 12408 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:38819
I20260923 22:17:40.624019 12485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:38819 every 8 connection(s)
I20260923 22:17:40.624388 12408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:40.625066 12486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:40.627388 12486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: Bootstrap starting.
I20260923 22:17:40.628005 12486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:40.628247 12486 log.cc:824] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:40.628854 12486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: No bootstrap required, opened a new log
I20260923 22:17:40.630913 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 12408
I20260923 22:17:40.630842 12486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bd61e55bd44451af58188af50b666f" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38819 } }
I20260923 22:17:40.630995 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:40.630997 12486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:40.631048 12486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bd61e55bd44451af58188af50b666f, State: Initialized, Role: FOLLOWER
I20260923 22:17:40.631204 4924 external_mini_cluster.cc:1428] Setting key 23169c87141c1f99631cdc2770d680ad
I20260923 22:17:40.631201 12486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bd61e55bd44451af58188af50b666f" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38819 } }
I20260923 22:17:40.631280 12486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:40.631323 12486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:40.631362 12486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:40.632055 12486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bd61e55bd44451af58188af50b666f" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38819 } }
I20260923 22:17:40.632143 12486 leader_election.cc:302] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36bd61e55bd44451af58188af50b666f; no voters:
I20260923 22:17:40.632321 12486 leader_election.cc:288] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:40.632347 12491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:40.632449 12491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [term 1 LEADER]: Becoming Leader. State: Replica: 36bd61e55bd44451af58188af50b666f, State: Running, Role: LEADER
I20260923 22:17:40.632539 12491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bd61e55bd44451af58188af50b666f" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38819 } }
I20260923 22:17:40.632577 12486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:40.632876 12491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36bd61e55bd44451af58188af50b666f. Latest consensus state: current_term: 1 leader_uuid: "36bd61e55bd44451af58188af50b666f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bd61e55bd44451af58188af50b666f" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38819 } } }
I20260923 22:17:40.632951 12491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:40.632895 12492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36bd61e55bd44451af58188af50b666f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bd61e55bd44451af58188af50b666f" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38819 } } }
I20260923 22:17:40.633112 12492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:40.633422 12496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:40.633917 12496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:40.635718 12496 catalog_manager.cc:1347] Generated new cluster ID: 72f0c3b56e0044b596239dd5dddebcb8
I20260923 22:17:40.635783 12496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:40 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:40.643064 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.631758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.4.207.62:38819)
0923 22:17:40.631954 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.631964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.632000 (+ 36us) server_negotiation.cc:407] Connection header received
0923 22:17:40.632093 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.632098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.632170 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.632323 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.633274 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.634690 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.635681 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.636274 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.636378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.639188 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.639192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.639192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.639204 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.641266 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.641792 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.641796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.641796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.641921 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.642125 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.642126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.642126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.642470 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.642639 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.642715 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.642857 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260923 22:17:40.648319 12496 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:40.648891 12496 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:40.652002 12496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: Generated new TSK 0
I20260923 22:17:40.652189 12496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab.
Entry for principal oryx/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab.
I20260923 22:17:40.678143 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38819
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab
--principal=oryx/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:40.761719 12512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.761960 12512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.761986 12512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.763270 12512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.763319 12512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.763340 12512 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:40.763357 12512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:40.764927 12512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab
--principal=oryx/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:38819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:40.765174 12512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.765367 12512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:40.770398 12512 server_base.cc:1034] running on GCE node
W20260923 22:17:40.770390 12517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.770507 12518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.770395 12520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.770833 12512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.771081 12512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.772238 12512 hybrid_clock.cc:648] HybridClock initialized: now 1790201860772225 us; error 29 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-7511 krb5kdc[12395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.774677 12512 init.cc:377] Logged in from keytab as oryx/127.4.207.1@KRBTEST.COM (short username oryx)
I20260923 22:17:40.775120 12512 webserver.cc:460] Webserver started at http://127.4.207.1:37127/ using document root <none> and password file <none>
I20260923 22:17:40.775357 12512 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.775420 12512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.775584 12512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:40.776644 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "d0aee12b8e784b7c843b24e35062716a"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "021117d1265fe091eccdde7567220244"
server_key_iv: "a866ae30ce160902475b04ea7b8a5491"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.777010 12512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "d0aee12b8e784b7c843b24e35062716a"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "021117d1265fe091eccdde7567220244"
server_key_iv: "a866ae30ce160902475b04ea7b8a5491"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.778301 12512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:40.779066 12527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.779232 12512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:40.779325 12512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "d0aee12b8e784b7c843b24e35062716a"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "021117d1265fe091eccdde7567220244"
server_key_iv: "a866ae30ce160902475b04ea7b8a5491"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.779402 12512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.802609 12512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.804678 12512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.804836 12512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.805353 12512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:40.805413 12512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.805465 12512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:40.805500 12512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.812318 12512 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:46735
I20260923 22:17:40.812394 12642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:46735 every 8 connection(s)
Sep 23 22:17:40 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:40.812696 12512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:40.813153 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 12512
I20260923 22:17:40.813284 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:40.813501 4924 external_mini_cluster.cc:1428] Setting key 283b3dfb0c75cabbc6e7f45f4d08286e
I20260923 22:17:40.816427 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.805899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:42601 (local address 127.4.207.62:38819)
0923 22:17:40.805943 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.805945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.806203 (+ 258us) server_negotiation.cc:407] Connection header received
0923 22:17:40.806683 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.806685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.806699 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.806765 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.807750 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.808636 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.810245 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.810594 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.810678 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.813254 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.813256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.813257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.813270 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.814853 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.815369 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.815370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.815371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.815468 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.815731 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.815732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.815732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.815885 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.815960 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.816217 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.816314 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:40.816464 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.805902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38819 (local address 127.4.207.1:42601)
0923 22:17:40.806068 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.806104 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.806592 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.806870 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.806874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.807291 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.807646 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.807654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.808739 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.808740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.810145 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.810148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.810682 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.810683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.810781 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.811343 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.811360 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.813134 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:40.814932 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.814939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.814941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.815279 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.815534 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.815535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.815535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.815666 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.815954 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:40.815958 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:40.816074 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 22:17:40.816183 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 22:17:40.816308 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260923 22:17:40.816985 12643 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38819
I20260923 22:17:40.817071 12643 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:40.817269 12643 heartbeater.cc:511] Master 127.4.207.62:38819 requested a full tablet report, sending...
I20260923 22:17:40.817803 12440 ts_manager.cc:194] Registered new tserver with Master: d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735)
I20260923 22:17:40.818785 12440 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:42601
WARNING: no policy specified for oryx/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.2@KRBTEST.COM" created.
I20260923 22:17:40.825708 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.819480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38819 (local address 127.4.207.1:36781)
0923 22:17:40.819638 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.819649 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.819714 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.819866 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.819866 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.820093 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.820190 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.820192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.820853 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.820854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.821733 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.821737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.822099 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.822100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.822198 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.822648 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.822654 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.823075 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:40.824656 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.824657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.824657 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.824823 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.825031 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.825032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.825032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.825118 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.825429 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:40.825430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:40.825498 (+ 68us) client_negotiation.cc:769] Sending connection context
0923 22:17:40.825556 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:17:40.825633 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260923 22:17:40.825827 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.819495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:36781 (local address 127.4.207.62:38819)
0923 22:17:40.819544 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.819546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.819713 (+ 167us) server_negotiation.cc:407] Connection header received
0923 22:17:40.819776 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.819777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.819787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.819854 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.820283 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.820751 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.821837 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.822020 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.822078 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.823152 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.823153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.823154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.823161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.824559 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.824888 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.824889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.824889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.824936 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.825236 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.825237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.825237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.825364 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.825418 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.825605 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.825694 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab.
Entry for principal oryx/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab.
I20260923 22:17:40.840590 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38819
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab
--principal=oryx/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:40.921048 12652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:40.921304 12652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:40.921343 12652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:40.922534 12652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:40.922606 12652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:40.922640 12652 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:40.922686 12652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:40.924273 12652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab
--principal=oryx/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:38819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:40.924551 12652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:40.924794 12652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:17:40.930068 12652 server_base.cc:1034] running on GCE node
W20260923 22:17:40.930040 12657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.930066 12660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:40.930172 12658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:40.930505 12652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:40.930745 12652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:40.931885 12652 hybrid_clock.cc:648] HybridClock initialized: now 1790201860931871 us; error 31 us; skew 500 ppm
Sep 23 22:17:40 dist-test-slave-7511 krb5kdc[12395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:40.934543 12652 init.cc:377] Logged in from keytab as oryx/127.4.207.2@KRBTEST.COM (short username oryx)
I20260923 22:17:40.935009 12652 webserver.cc:460] Webserver started at http://127.4.207.2:45429/ using document root <none> and password file <none>
I20260923 22:17:40.935209 12652 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:40.935251 12652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:40.935343 12652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:40.936420 12652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "b453b4f3a8694760bff9c14f49956eab"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "b7c7c8b6a58bb0a49d4663c56445374f"
server_key_iv: "06461b2e5c20f7e9a7dfb059a658b000"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.936735 12652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "b453b4f3a8694760bff9c14f49956eab"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "b7c7c8b6a58bb0a49d4663c56445374f"
server_key_iv: "06461b2e5c20f7e9a7dfb059a658b000"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.937999 12652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:17:40.938717 12667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.938890 12652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:40.938990 12652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "b453b4f3a8694760bff9c14f49956eab"
format_stamp: "Formatted at 2026-09-23 22:17:40 on dist-test-slave-7511"
server_key: "b7c7c8b6a58bb0a49d4663c56445374f"
server_key_iv: "06461b2e5c20f7e9a7dfb059a658b000"
server_key_version: "encryptionkey@0"
I20260923 22:17:40.939064 12652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:40.956881 12652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:40.958918 12652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:40.959038 12652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:40.959640 12652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:40.959683 12652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.959708 12652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:40.959721 12652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:40.966674 12652 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:34767
I20260923 22:17:40.966773 12782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:34767 every 8 connection(s)
I20260923 22:17:40.967089 12652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:40 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:40.970623 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.960117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:44033 (local address 127.4.207.62:38819)
0923 22:17:40.960187 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.960190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.960404 (+ 214us) server_negotiation.cc:407] Connection header received
0923 22:17:40.960947 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.960948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.960960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.960998 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.961865 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.962703 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.964547 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.964834 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.964892 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.967887 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.967889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.967889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.967897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.969410 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.969843 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.969844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.969845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.969894 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.970105 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.970105 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.970106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.970212 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.970261 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.970496 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.970558 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:17:40.970829 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.960090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38819 (local address 127.4.207.2:44033)
0923 22:17:40.960251 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.960291 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.960831 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.961070 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.961074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.961478 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.961775 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.961782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.962810 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.962811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.964426 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.964430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.964922 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.964924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.965103 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.965742 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.965754 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.967768 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:40.969504 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.969511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.969513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.969749 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.969951 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.969952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.969952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.970052 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.970265 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:40.970269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:40.970389 (+ 120us) client_negotiation.cc:769] Sending connection context
0923 22:17:40.970473 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:17:40.970597 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260923 22:17:40.971495 12783 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38819
I20260923 22:17:40.971602 12783 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:40.971788 12783 heartbeater.cc:511] Master 127.4.207.62:38819 requested a full tablet report, sending...
I20260923 22:17:40.972229 12440 ts_manager.cc:194] Registered new tserver with Master: b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767)
I20260923 22:17:40.972683 12440 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:44033
I20260923 22:17:40.975606 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 12652
I20260923 22:17:40.975713 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:40.975931 4924 external_mini_cluster.cc:1428] Setting key 9dede29c8fa19a8eb76c49ef4e6f1d65
I20260923 22:17:40.978432 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.972564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:46795 (local address 127.4.207.62:38819)
0923 22:17:40.972619 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:40.972621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:40.972749 (+ 128us) server_negotiation.cc:407] Connection header received
0923 22:17:40.972829 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:40.972830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:40.972842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:40.972913 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:40.973220 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.973705 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.974437 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.974642 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.974711 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.975779 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:40.975781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:40.975781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:40.975790 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:40.977260 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.977614 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.977615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.977615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.977674 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:40.977863 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:40.977864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:40.977864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:40.978013 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:40.978086 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:40.978243 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 22:17:40.978326 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:40.978446 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:40.972506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38819 (local address 127.4.207.2:46795)
0923 22:17:40.972676 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:40.972691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:40.972768 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:40.972923 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:40.972924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:40.973087 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:40.973146 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.973147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.973800 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.973801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.974342 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:40.974343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:40.974720 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:40.974721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:40.974796 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:40.975238 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:40.975242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:40.975649 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:40.977365 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.977366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.977367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.977543 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.977736 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:40.977737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:40.977737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:40.977796 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:40.978089 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:40.978090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:40.978138 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 22:17:40.978219 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 22:17:40.978328 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab.
Entry for principal oryx/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab.
I20260923 22:17:41.001564 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38819
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab
--principal=oryx/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:41.084329 12792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:41.084576 12792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:41.084618 12792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:41.085785 12792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:41.085851 12792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:41.085884 12792 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 22:17:41.085922 12792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:41.087421 12792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab
--principal=oryx/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.207.62:38819
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:41.087702 12792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:41.087949 12792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:41.093114 12800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:41.093168 12798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:41.093194 12797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:41.093487 12792 server_base.cc:1034] running on GCE node
I20260923 22:17:41.093690 12792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:41.093940 12792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:41.095108 12792 hybrid_clock.cc:648] HybridClock initialized: now 1790201861095102 us; error 37 us; skew 500 ppm
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:41.097839 12792 init.cc:377] Logged in from keytab as oryx/127.4.207.3@KRBTEST.COM (short username oryx)
I20260923 22:17:41.098320 12792 webserver.cc:460] Webserver started at http://127.4.207.3:41305/ using document root <none> and password file <none>
I20260923 22:17:41.098546 12792 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:41.098609 12792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:41.098753 12792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:41.099908 12792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "26f8c19a146643518dcc6e64519b7f7f"
format_stamp: "Formatted at 2026-09-23 22:17:41 on dist-test-slave-7511"
server_key: "eaa915a00e31e49ea7205eca58e2ea72"
server_key_iv: "be2854b1b1bed07b516569002a42abbe"
server_key_version: "encryptionkey@0"
I20260923 22:17:41.100265 12792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "26f8c19a146643518dcc6e64519b7f7f"
format_stamp: "Formatted at 2026-09-23 22:17:41 on dist-test-slave-7511"
server_key: "eaa915a00e31e49ea7205eca58e2ea72"
server_key_iv: "be2854b1b1bed07b516569002a42abbe"
server_key_version: "encryptionkey@0"
I20260923 22:17:41.101544 12792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 22:17:41.102289 12807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:41.102455 12792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:41.102553 12792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "26f8c19a146643518dcc6e64519b7f7f"
format_stamp: "Formatted at 2026-09-23 22:17:41 on dist-test-slave-7511"
server_key: "eaa915a00e31e49ea7205eca58e2ea72"
server_key_iv: "be2854b1b1bed07b516569002a42abbe"
server_key_version: "encryptionkey@0"
I20260923 22:17:41.102625 12792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:41.110826 12792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:41.112912 12792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:41.113075 12792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:41.113628 12792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:41.113683 12792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:41.113725 12792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:41.113766 12792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:41.121114 12792 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:33283
I20260923 22:17:41.121493 12792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:41.121457 12922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:33283 every 8 connection(s)
I20260923 22:17:41.123433 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.114108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:35829 (local address 127.4.207.62:38819)
0923 22:17:41.114182 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.114184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.114379 (+ 195us) server_negotiation.cc:407] Connection header received
0923 22:17:41.114866 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.114868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.114883 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.114969 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.115707 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.116218 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.117370 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.117624 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.117709 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.120383 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:41.120385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:41.120386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:41.120395 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:41.122036 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.122472 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.122473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.122474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.122530 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.122775 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.122776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.122777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.122898 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:41.122974 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.123244 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.123345 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:17:41.123641 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.114083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38819 (local address 127.4.207.3:35829)
0923 22:17:41.114226 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.114266 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.114773 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.115022 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.115026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.115354 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:41.115614 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.115620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.116313 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.116314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.117174 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.117176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.117719 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.117720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.117851 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.118387 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:41.118395 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:41.120285 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:41.122128 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:41.122133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:41.122135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:41.122384 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:41.122598 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:41.122599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:41.122600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:41.122712 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:41.122964 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:41.122970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:41.123112 (+ 142us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.123221 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.123382 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
I20260923 22:17:41.124089 12923 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38819
I20260923 22:17:41.124185 12923 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:41.124365 12923 heartbeater.cc:511] Master 127.4.207.62:38819 requested a full tablet report, sending...
I20260923 22:17:41.124775 12439 ts_manager.cc:194] Registered new tserver with Master: 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283)
I20260923 22:17:41.125254 12439 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:35829
I20260923 22:17:41.126626 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 12792
I20260923 22:17:41.126750 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:41.126957 4924 external_mini_cluster.cc:1428] Setting key c0833f8a241bceb48d0a74e072c8c058
I20260923 22:17:41.127892 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:17:41.131317 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.125106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38819 (local address 127.4.207.3:33883)
0923 22:17:41.125228 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.125240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.125312 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.125468 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.125469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.125656 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:41.125714 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.125715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.126370 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.126370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.127246 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.127248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.127699 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.127700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.127768 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.128245 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:41.128251 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:41.128541 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:41.130374 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:41.130375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:41.130375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:41.130540 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:41.130742 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:41.130743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:41.130743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:41.130804 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:41.131070 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:41.131070 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:41.131111 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.131163 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.131244 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260923 22:17:41.131407 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.125192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:33883 (local address 127.4.207.62:38819)
0923 22:17:41.125233 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.125235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.125328 (+ 93us) server_negotiation.cc:407] Connection header received
0923 22:17:41.125361 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.125362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.125372 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.125426 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.125800 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.126272 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.127321 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.127618 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.127689 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.128645 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:41.128646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:41.128646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:41.128653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:41.130287 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.130614 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.130615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.130616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.130679 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.130863 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.130864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.130864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.131006 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:41.131083 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.131195 (+ 112us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.131278 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 22:17:41.150301 12490 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:41.144970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.4.207.62:38819)
0923 22:17:41.145033 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.145036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.145044 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:41.145074 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.145075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.145083 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.145146 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.145692 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.146200 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.146954 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.147142 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.147214 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.150194 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38974: BlockingRecv error: recv got EOF from 127.0.0.1:38974 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201861, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:41.158020 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.151224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.4.207.62:38819)
0923 22:17:41.151294 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.151296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.151389 (+ 93us) server_negotiation.cc:407] Connection header received
0923 22:17:41.151498 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.151498 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.151508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.151568 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.152041 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.152622 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.153354 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.153544 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.153623 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.155754 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:41.155755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:41.155756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:41.155763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:41.157141 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.157429 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.157430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.157431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.157480 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.157617 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.157618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.157618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.157714 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:41.157782 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.157859 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.157917 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:17:41.159711 12439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:41.161259 12439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:41.168052 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.163873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.4.207.2:34767)
0923 22:17:41.164297 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.164301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.164310 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:41.164346 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.164348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.164366 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.164455 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:41.164647 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.165454 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.166764 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.167559 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.167635 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.167709 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.167900 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.167940 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":77,"threads_started":1}
I20260923 22:17:41.168061 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.163759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:34767 (local address 127.0.0.1:59306)
0923 22:17:41.164068 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.164239 (+ 171us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.164275 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.164475 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.164479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.164481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:41.164550 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.164552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.165531 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.165532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.166658 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.166660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.167645 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.167646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.167746 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.167764 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.167877 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.167940 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260923 22:17:41.168071 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.163759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33283 (local address 127.0.0.1:53290)
0923 22:17:41.164042 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.164088 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.164164 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.164532 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.164534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.164537 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:41.164588 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.164590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.165963 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.165964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.166779 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.166781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.167763 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.167765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.167828 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.167833 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.167907 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.167961 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":40,"thread_start_us":59,"threads_started":1}
I20260923 22:17:41.168233 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.163963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:46735 (local address 127.0.0.1:40450)
0923 22:17:41.164116 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.164123 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.164164 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.164724 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.164725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.164728 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:41.164762 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.164763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.165964 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.165965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.167101 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.167103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.168021 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.168021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.168078 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.168082 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.168115 (+ 33us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.168150 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":31,"thread_start_us":74,"threads_started":1}
I20260923 22:17:41.168687 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.164074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40450 (local address 127.4.207.1:46735)
0923 22:17:41.164580 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.164584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.164593 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:41.164629 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.164631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.164646 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.164728 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:41.164819 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.165865 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.167183 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.167931 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.168428 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.168515 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.168540 (+ 25us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.168578 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":84,"threads_started":1}
I20260923 22:17:41.168991 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.163859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.4.207.3:33283)
0923 22:17:41.164184 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.164189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.164202 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:17:41.164369 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.164373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.164391 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.164459 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:41.164685 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.165860 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.166862 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.167651 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.167729 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.167807 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.168882 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.168920 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":94,"threads_started":1}
I20260923 22:17:41.168949 12717 tablet_service.cc:1467] Processing CreateTablet for tablet 9053f384e19e4dfd878686dc4de7cd9c (DEFAULT_TABLE table=test-table [id=86c5b03443cd437a97d0bd42bc10e38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.169370 12717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9053f384e19e4dfd878686dc4de7cd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.169315 12577 tablet_service.cc:1467] Processing CreateTablet for tablet 9053f384e19e4dfd878686dc4de7cd9c (DEFAULT_TABLE table=test-table [id=86c5b03443cd437a97d0bd42bc10e38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.169677 12577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9053f384e19e4dfd878686dc4de7cd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.169667 12857 tablet_service.cc:1467] Processing CreateTablet for tablet 9053f384e19e4dfd878686dc4de7cd9c (DEFAULT_TABLE table=test-table [id=86c5b03443cd437a97d0bd42bc10e38b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.169988 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9053f384e19e4dfd878686dc4de7cd9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.172991 12951 tablet_bootstrap.cc:492] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: Bootstrap starting.
I20260923 22:17:41.173477 12952 tablet_bootstrap.cc:492] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: Bootstrap starting.
I20260923 22:17:41.173750 12951 tablet_bootstrap.cc:654] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.174110 12952 tablet_bootstrap.cc:654] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.174089 12951 log.cc:824] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:41.174427 12952 log.cc:824] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:41.174752 12953 tablet_bootstrap.cc:492] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: Bootstrap starting.
I20260923 22:17:41.174917 12951 tablet_bootstrap.cc:492] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: No bootstrap required, opened a new log
I20260923 22:17:41.174988 12951 ts_tablet_manager.cc:1397] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:41.175128 12952 tablet_bootstrap.cc:492] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: No bootstrap required, opened a new log
I20260923 22:17:41.175205 12952 ts_tablet_manager.cc:1397] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:41.175359 12953 tablet_bootstrap.cc:654] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.175738 12953 log.cc:824] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:41.176478 12953 tablet_bootstrap.cc:492] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: No bootstrap required, opened a new log
I20260923 22:17:41.176544 12953 ts_tablet_manager.cc:1397] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:41.177245 12952 raft_consensus.cc:348] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.177239 12951 raft_consensus.cc:348] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.177397 12951 raft_consensus.cc:374] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.177398 12952 raft_consensus.cc:374] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.177443 12952 raft_consensus.cc:729] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b453b4f3a8694760bff9c14f49956eab, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.177443 12951 raft_consensus.cc:729] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f8c19a146643518dcc6e64519b7f7f, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.177579 12952 consensus_queue.cc:260] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.177579 12951 consensus_queue.cc:260] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.177865 12952 ts_tablet_manager.cc:1428] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:41.177882 12923 heartbeater.cc:503] Master 127.4.207.62:38819 was elected leader, sending a full tablet report...
I20260923 22:17:41.177989 12783 heartbeater.cc:503] Master 127.4.207.62:38819 was elected leader, sending a full tablet report...
I20260923 22:17:41.178154 12951 ts_tablet_manager.cc:1428] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:17:41.178570 12953 raft_consensus.cc:348] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.178762 12953 raft_consensus.cc:374] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.178812 12953 raft_consensus.cc:729] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aee12b8e784b7c843b24e35062716a, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.178926 12953 consensus_queue.cc:260] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.179181 12953 ts_tablet_manager.cc:1428] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:17:41.179239 12643 heartbeater.cc:503] Master 127.4.207.62:38819 was elected leader, sending a full tablet report...
I20260923 22:17:41.189453 12959 raft_consensus.cc:482] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:41.189602 12959 raft_consensus.cc:504] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.190047 12959 leader_election.cc:288] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283), b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767)
I20260923 22:17:41.193073 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.190299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:54549 (local address 127.4.207.3:33283)
0923 22:17:41.190353 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.190355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.190365 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:41.190395 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.190396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.190408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.190475 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:41.190640 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.191168 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.192358 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.192707 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.192764 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.192772 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.192949 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.192997 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:17:41.193097 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.190315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:47035 (local address 127.4.207.2:34767)
0923 22:17:41.190360 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.190362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.190471 (+ 109us) server_negotiation.cc:407] Connection header received
0923 22:17:41.190531 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.190532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.190545 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.190617 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:41.190822 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.191370 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.192357 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.192724 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.192813 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.192825 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.192983 (+ 158us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.193026 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:41.193097 12536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.190201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33283 (local address 127.4.207.1:54549)
0923 22:17:41.190255 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.190268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.190310 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.190485 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.190486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.190487 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:41.190562 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.190564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.191253 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.191255 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.192266 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.192269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.192782 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.192783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.192882 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.192885 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.192928 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.192990 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":29}
I20260923 22:17:41.193552 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9053f384e19e4dfd878686dc4de7cd9c" candidate_uuid: "d0aee12b8e784b7c843b24e35062716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f8c19a146643518dcc6e64519b7f7f" is_pre_election: true
I20260923 22:17:41.193603 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.190215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:34767 (local address 127.4.207.1:47035)
0923 22:17:41.190400 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.190411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.190468 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.190639 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.190640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.190641 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:41.190715 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.190716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.191494 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.191495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.192271 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.192273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.192830 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.192831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.192897 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.192900 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.192989 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.193026 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":108,"threads_started":1}
I20260923 22:17:41.193725 12873 raft_consensus.cc:2454] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aee12b8e784b7c843b24e35062716a in term 0.
I20260923 22:17:41.193974 12737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9053f384e19e4dfd878686dc4de7cd9c" candidate_uuid: "d0aee12b8e784b7c843b24e35062716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b453b4f3a8694760bff9c14f49956eab" is_pre_election: true
I20260923 22:17:41.194061 12531 leader_election.cc:302] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f8c19a146643518dcc6e64519b7f7f, d0aee12b8e784b7c843b24e35062716a; no voters:
I20260923 22:17:41.194130 12737 raft_consensus.cc:2454] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aee12b8e784b7c843b24e35062716a in term 0.
I20260923 22:17:41.194221 12959 raft_consensus.cc:2781] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:41.194294 12959 raft_consensus.cc:482] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:41.194321 12959 raft_consensus.cc:3037] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.195338 12959 raft_consensus.cc:504] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.195595 12959 leader_election.cc:288] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [CANDIDATE]: Term 1 election: Requested vote from peers 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283), b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767)
I20260923 22:17:41.195662 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9053f384e19e4dfd878686dc4de7cd9c" candidate_uuid: "d0aee12b8e784b7c843b24e35062716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f8c19a146643518dcc6e64519b7f7f"
I20260923 22:17:41.195735 12737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9053f384e19e4dfd878686dc4de7cd9c" candidate_uuid: "d0aee12b8e784b7c843b24e35062716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b453b4f3a8694760bff9c14f49956eab"
I20260923 22:17:41.195755 12873 raft_consensus.cc:3037] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.195829 12737 raft_consensus.cc:3037] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.196683 12873 raft_consensus.cc:2454] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aee12b8e784b7c843b24e35062716a in term 1.
I20260923 22:17:41.196693 12737 raft_consensus.cc:2454] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aee12b8e784b7c843b24e35062716a in term 1.
I20260923 22:17:41.196836 12531 leader_election.cc:302] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f8c19a146643518dcc6e64519b7f7f, d0aee12b8e784b7c843b24e35062716a; no voters:
I20260923 22:17:41.196941 12959 raft_consensus.cc:2781] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:41.197134 12959 raft_consensus.cc:686] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 1 LEADER]: Becoming Leader. State: Replica: d0aee12b8e784b7c843b24e35062716a, State: Running, Role: LEADER
I20260923 22:17:41.197240 12959 consensus_queue.cc:237] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } }
I20260923 22:17:41.198112 12439 catalog_manager.cc:5495] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a reported cstate change: term changed from 0 to 1, leader changed from <none> to d0aee12b8e784b7c843b24e35062716a (127.4.207.1). New cstate: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:41.208628 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.206233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.4.207.1:46735)
0923 22:17:41.206285 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.206287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.206302 (+ 15us) server_negotiation.cc:407] Connection header received
0923 22:17:41.206360 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.206361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.206380 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.206444 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:41.206568 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.207077 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.207951 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.208149 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.208212 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.208360 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:41.208466 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:41.208530 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.208549 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.208566 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:17:41.210537 12737 raft_consensus.cc:1261] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Refusing update from remote peer d0aee12b8e784b7c843b24e35062716a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:41.210566 12873 raft_consensus.cc:1261] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Refusing update from remote peer d0aee12b8e784b7c843b24e35062716a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:41.210879 12961 consensus_queue.cc:1035] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:41.211015 12959 consensus_queue.cc:1035] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:41.213430 12964 mvcc.cc:205] Tried to move back new op lower bound from 7332666823515451392 to 7332666823464640512. Current Snapshot: MvccSnapshot[applied={T|T < 7332666823515451392}]
I20260923 22:17:41.213814 12966 mvcc.cc:205] Tried to move back new op lower bound from 7332666823515451392 to 7332666823464640512. Current Snapshot: MvccSnapshot[applied={T|T < 7332666823515451392}]
I20260923 22:17:41.216367 12440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:41.216470 12440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:41.217262 12440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: Sending DeleteTablet for 3 replicas of tablet 9053f384e19e4dfd878686dc4de7cd9c
I20260923 22:17:41.217675 12717 tablet_service.cc:1514] Processing DeleteTablet for tablet 9053f384e19e4dfd878686dc4de7cd9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:41 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:59306
I20260923 22:17:41.217705 12577 tablet_service.cc:1514] Processing DeleteTablet for tablet 9053f384e19e4dfd878686dc4de7cd9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:41 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:40450
I20260923 22:17:41.217942 12857 tablet_service.cc:1514] Processing DeleteTablet for tablet 9053f384e19e4dfd878686dc4de7cd9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:41 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:53290
I20260923 22:17:41.218272 12975 tablet_replica.cc:330] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: stopping tablet replica
I20260923 22:17:41.218236 12440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:41.218387 12440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:41.218401 12975 raft_consensus.cc:2229] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:41.218528 12975 raft_consensus.cc:2258] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:41.218706 12976 tablet_replica.cc:330] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: stopping tablet replica
I20260923 22:17:41.218815 12976 raft_consensus.cc:2229] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:41.218938 12976 raft_consensus.cc:2258] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:41.219151 12975 ts_tablet_manager.cc:1905] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:41.219458 12976 ts_tablet_manager.cc:1905] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:41.219779 12717 tablet_service.cc:1467] Processing CreateTablet for tablet 114611b3577045a29296466f0acf1f33 (DEFAULT_TABLE table=test-table [id=7b97a03cba0542de90ced95b4ea2d8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.219894 12717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114611b3577045a29296466f0acf1f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.219957 12857 tablet_service.cc:1467] Processing CreateTablet for tablet 114611b3577045a29296466f0acf1f33 (DEFAULT_TABLE table=test-table [id=7b97a03cba0542de90ced95b4ea2d8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.220060 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114611b3577045a29296466f0acf1f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.221065 12975 ts_tablet_manager.cc:1918] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:41.221125 12975 log.cc:1196] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/9053f384e19e4dfd878686dc4de7cd9c
I20260923 22:17:41.221380 12975 ts_tablet_manager.cc:1939] T 9053f384e19e4dfd878686dc4de7cd9c P b453b4f3a8694760bff9c14f49956eab: Deleting consensus metadata
I20260923 22:17:41.221561 12976 ts_tablet_manager.cc:1918] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:41.221665 12952 tablet_bootstrap.cc:492] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: Bootstrap starting.
I20260923 22:17:41.221664 12951 tablet_bootstrap.cc:492] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: Bootstrap starting.
I20260923 22:17:41.222029 12951 tablet_bootstrap.cc:654] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.222069 12952 tablet_bootstrap.cc:654] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.221655 12976 log.cc:1196] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/9053f384e19e4dfd878686dc4de7cd9c
I20260923 22:17:41.222503 12425 catalog_manager.cc:4841] TS b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767): tablet 9053f384e19e4dfd878686dc4de7cd9c (table test-table [id=86c5b03443cd437a97d0bd42bc10e38b]) successfully deleted
I20260923 22:17:41.222806 12951 tablet_bootstrap.cc:492] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: No bootstrap required, opened a new log
I20260923 22:17:41.222832 12577 tablet_service.cc:1467] Processing CreateTablet for tablet 114611b3577045a29296466f0acf1f33 (DEFAULT_TABLE table=test-table [id=7b97a03cba0542de90ced95b4ea2d8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.222860 12951 ts_tablet_manager.cc:1397] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:41.222944 12577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 114611b3577045a29296466f0acf1f33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.223011 12951 raft_consensus.cc:348] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.223114 12951 raft_consensus.cc:374] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.223148 12951 raft_consensus.cc:729] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f8c19a146643518dcc6e64519b7f7f, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.223199 12951 consensus_queue.cc:260] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.223309 12951 ts_tablet_manager.cc:1428] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:41.223852 12976 ts_tablet_manager.cc:1939] T 9053f384e19e4dfd878686dc4de7cd9c P 26f8c19a146643518dcc6e64519b7f7f: Deleting consensus metadata
I20260923 22:17:41.224303 12953 tablet_bootstrap.cc:492] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: Bootstrap starting.
I20260923 22:17:41.224327 12427 catalog_manager.cc:4841] TS 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283): tablet 9053f384e19e4dfd878686dc4de7cd9c (table test-table [id=86c5b03443cd437a97d0bd42bc10e38b]) successfully deleted
I20260923 22:17:41.224522 12952 tablet_bootstrap.cc:492] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: No bootstrap required, opened a new log
I20260923 22:17:41.224577 12952 ts_tablet_manager.cc:1397] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 22:17:41.224748 12953 tablet_bootstrap.cc:654] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.224721 12952 raft_consensus.cc:348] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.224800 12952 raft_consensus.cc:374] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.224838 12952 raft_consensus.cc:729] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b453b4f3a8694760bff9c14f49956eab, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.224877 12952 consensus_queue.cc:260] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.224958 12952 ts_tablet_manager.cc:1428] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:41.227583 12974 tablet_replica.cc:330] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: stopping tablet replica
I20260923 22:17:41.227718 12974 raft_consensus.cc:2229] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:41.227877 12974 raft_consensus.cc:2258] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:41.228307 12974 ts_tablet_manager.cc:1905] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:41.228322 12953 tablet_bootstrap.cc:492] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: No bootstrap required, opened a new log
I20260923 22:17:41.228397 12953 ts_tablet_manager.cc:1397] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 22:17:41.228551 12953 raft_consensus.cc:348] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.228647 12953 raft_consensus.cc:374] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.228668 12953 raft_consensus.cc:729] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aee12b8e784b7c843b24e35062716a, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.228716 12953 consensus_queue.cc:260] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.228811 12953 ts_tablet_manager.cc:1428] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:41.230144 12974 ts_tablet_manager.cc:1918] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:41.230203 12974 log.cc:1196] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/9053f384e19e4dfd878686dc4de7cd9c
I20260923 22:17:41.230496 12974 ts_tablet_manager.cc:1939] T 9053f384e19e4dfd878686dc4de7cd9c P d0aee12b8e784b7c843b24e35062716a: Deleting consensus metadata
I20260923 22:17:41.230906 12427 catalog_manager.cc:4841] TS d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735): tablet 9053f384e19e4dfd878686dc4de7cd9c (table test-table [id=86c5b03443cd437a97d0bd42bc10e38b]) successfully deleted
I20260923 22:17:41.244724 12957 raft_consensus.cc:482] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:41.244855 12957 raft_consensus.cc:504] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.245250 12957 leader_election.cc:288] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735), b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767)
I20260923 22:17:41.248631 12816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.245417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:46735 (local address 127.4.207.3:33977)
0923 22:17:41.245472 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.245485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.245527 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.245678 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.245680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.245681 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:41.245760 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.245762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.246476 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.246476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.247618 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.247621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.248403 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.248405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.248484 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.248487 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.248532 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.248566 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 22:17:41.248661 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.245491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:33977 (local address 127.4.207.1:46735)
0923 22:17:41.245537 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.245539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.245551 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:41.245580 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.245581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.245591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.245639 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:41.245842 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.246387 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.247706 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.248311 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.248393 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.248405 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.248553 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.248588 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:17:41.248919 12597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114611b3577045a29296466f0acf1f33" candidate_uuid: "26f8c19a146643518dcc6e64519b7f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aee12b8e784b7c843b24e35062716a" is_pre_election: true
I20260923 22:17:41.249069 12597 raft_consensus.cc:2454] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f8c19a146643518dcc6e64519b7f7f in term 0.
I20260923 22:17:41.249135 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.245605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:52193 (local address 127.4.207.2:34767)
0923 22:17:41.245648 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.245650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.245779 (+ 129us) server_negotiation.cc:407] Connection header received
0923 22:17:41.245831 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.245832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.245840 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.245901 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:41.246029 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.246770 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.247971 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.248326 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.248414 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.248424 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.248993 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.249060 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:17:41.249280 12811 leader_election.cc:302] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f8c19a146643518dcc6e64519b7f7f, d0aee12b8e784b7c843b24e35062716a; no voters:
I20260923 22:17:41.249445 12957 raft_consensus.cc:2781] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:41.249528 12957 raft_consensus.cc:482] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:41.249554 12957 raft_consensus.cc:3037] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.249562 12980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.245419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:34767 (local address 127.4.207.3:52193)
0923 22:17:41.245707 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.245717 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.245772 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.245900 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.245901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.245902 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:41.245969 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.245970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.246860 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.246861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.247891 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.247893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.248845 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.248846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.248912 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.248915 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.249236 (+ 321us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.249280 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":13,"thread_start_us":77,"threads_started":1}
I20260923 22:17:41.249806 12737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114611b3577045a29296466f0acf1f33" candidate_uuid: "26f8c19a146643518dcc6e64519b7f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b453b4f3a8694760bff9c14f49956eab" is_pre_election: true
I20260923 22:17:41.249891 12737 raft_consensus.cc:2454] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f8c19a146643518dcc6e64519b7f7f in term 0.
I20260923 22:17:41.250510 12957 raft_consensus.cc:504] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.250655 12957 leader_election.cc:288] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [CANDIDATE]: Term 1 election: Requested vote from peers d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735), b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767)
I20260923 22:17:41.250797 12737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114611b3577045a29296466f0acf1f33" candidate_uuid: "26f8c19a146643518dcc6e64519b7f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b453b4f3a8694760bff9c14f49956eab"
I20260923 22:17:41.250864 12737 raft_consensus.cc:3037] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.251536 12597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "114611b3577045a29296466f0acf1f33" candidate_uuid: "26f8c19a146643518dcc6e64519b7f7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aee12b8e784b7c843b24e35062716a"
I20260923 22:17:41.251610 12597 raft_consensus.cc:3037] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.251710 12737 raft_consensus.cc:2454] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f8c19a146643518dcc6e64519b7f7f in term 1.
I20260923 22:17:41.251888 12809 leader_election.cc:302] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f8c19a146643518dcc6e64519b7f7f, b453b4f3a8694760bff9c14f49956eab; no voters:
I20260923 22:17:41.251982 12957 raft_consensus.cc:2781] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:41.252136 12957 raft_consensus.cc:686] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 1 LEADER]: Becoming Leader. State: Replica: 26f8c19a146643518dcc6e64519b7f7f, State: Running, Role: LEADER
I20260923 22:17:41.252202 12957 consensus_queue.cc:237] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.252509 12597 raft_consensus.cc:2454] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f8c19a146643518dcc6e64519b7f7f in term 1.
I20260923 22:17:41.252903 12439 catalog_manager.cc:5495] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3). New cstate: current_term: 1 leader_uuid: "26f8c19a146643518dcc6e64519b7f7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } health_report { overall_health: HEALTHY } } }
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.62@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:41.287981 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.280238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38819 (local address 127.0.0.1:38994)
0923 22:17:41.280431 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.280445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.280506 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.280655 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.280656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.280947 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:41.280995 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.280997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.281982 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.281984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.282552 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.282554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.283009 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.283010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.283095 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.283727 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:41.283731 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:41.285061 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:41.286859 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:41.286865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:41.286867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:41.287080 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:41.287284 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:41.287285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:41.287285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:41.287347 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:41.287656 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:41.287662 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:41.287716 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.287777 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.287881 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
I20260923 22:17:41.288067 12490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.280333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.4.207.62:38819)
0923 22:17:41.280397 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.280400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.280507 (+ 107us) server_negotiation.cc:407] Connection header received
0923 22:17:41.280556 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.280557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.280569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.280650 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.281079 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.281872 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.282638 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.282927 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.283009 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.285152 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:41.285154 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:41.285154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:41.285162 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:41.286773 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.287154 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.287155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.287155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.287216 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.287407 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.287408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.287408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.287571 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:41.287638 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.287822 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.287947 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":35}
I20260923 22:17:41.290540 12439 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:38994:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 22:17:41.290735 12439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:41.292178 12857 tablet_service.cc:1467] Processing CreateTablet for tablet 4e80ef250a5b41d2bd39284ec9bf0197 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aba55388629b4cd3994fa7f6bab58ffe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:41.292174 12717 tablet_service.cc:1467] Processing CreateTablet for tablet 4e80ef250a5b41d2bd39284ec9bf0197 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aba55388629b4cd3994fa7f6bab58ffe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:41.292328 12717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e80ef250a5b41d2bd39284ec9bf0197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.292353 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e80ef250a5b41d2bd39284ec9bf0197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.293911 12951 tablet_bootstrap.cc:492] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: Bootstrap starting.
I20260923 22:17:41.293998 12577 tablet_service.cc:1467] Processing CreateTablet for tablet 4e80ef250a5b41d2bd39284ec9bf0197 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aba55388629b4cd3994fa7f6bab58ffe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 22:17:41.294147 12577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e80ef250a5b41d2bd39284ec9bf0197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.294307 12951 tablet_bootstrap.cc:654] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.294909 12951 tablet_bootstrap.cc:492] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: No bootstrap required, opened a new log
I20260923 22:17:41.294941 12951 ts_tablet_manager.cc:1397] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:41.295063 12951 raft_consensus.cc:348] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.295113 12951 raft_consensus.cc:374] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.295126 12951 raft_consensus.cc:729] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f8c19a146643518dcc6e64519b7f7f, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.295181 12951 consensus_queue.cc:260] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.295277 12951 ts_tablet_manager.cc:1428] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:41.295403 12957 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.295501 12957 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:41.295667 12953 tablet_bootstrap.cc:492] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: Bootstrap starting.
I20260923 22:17:41.295697 12952 tablet_bootstrap.cc:492] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: Bootstrap starting.
I20260923 22:17:41.296088 12953 tablet_bootstrap.cc:654] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.296128 12952 tablet_bootstrap.cc:654] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.296725 12953 tablet_bootstrap.cc:492] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: No bootstrap required, opened a new log
I20260923 22:17:41.296765 12953 ts_tablet_manager.cc:1397] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:41.297045 12953 raft_consensus.cc:348] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.297106 12952 tablet_bootstrap.cc:492] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: No bootstrap required, opened a new log
I20260923 22:17:41.297134 12953 raft_consensus.cc:374] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.297148 12952 ts_tablet_manager.cc:1397] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:41.297160 12953 raft_consensus.cc:729] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aee12b8e784b7c843b24e35062716a, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.297217 12953 consensus_queue.cc:260] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.297314 12953 ts_tablet_manager.cc:1428] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:41.297317 12952 raft_consensus.cc:348] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.297415 12952 raft_consensus.cc:374] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.297396 12959 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.297446 12952 raft_consensus.cc:729] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b453b4f3a8694760bff9c14f49956eab, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.297459 12959 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:41.297495 12952 consensus_queue.cc:260] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.297590 12952 ts_tablet_manager.cc:1428] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:41.297752 12958 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.297847 12958 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 22:17:41.313436 12644 tablet.cc:2369] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:41.372292 12924 tablet.cc:2369] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:17:41.467934 12784 tablet.cc:2369] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:17:41.638975 12957 consensus_queue.cc:1035] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:41.640977 12993 consensus_queue.cc:1035] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:41.648725 12959 raft_consensus.cc:482] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:41.648816 12959 raft_consensus.cc:504] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.648967 12959 leader_election.cc:288] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767), 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283)
I20260923 22:17:41.649176 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e80ef250a5b41d2bd39284ec9bf0197" candidate_uuid: "d0aee12b8e784b7c843b24e35062716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f8c19a146643518dcc6e64519b7f7f" is_pre_election: true
I20260923 22:17:41.649184 12737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e80ef250a5b41d2bd39284ec9bf0197" candidate_uuid: "d0aee12b8e784b7c843b24e35062716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b453b4f3a8694760bff9c14f49956eab" is_pre_election: true
I20260923 22:17:41.649271 12737 raft_consensus.cc:2454] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aee12b8e784b7c843b24e35062716a in term 0.
I20260923 22:17:41.649271 12873 raft_consensus.cc:2454] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0aee12b8e784b7c843b24e35062716a in term 0.
I20260923 22:17:41.649420 12531 leader_election.cc:302] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f8c19a146643518dcc6e64519b7f7f, d0aee12b8e784b7c843b24e35062716a; no voters:
I20260923 22:17:41.649545 12959 raft_consensus.cc:2781] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:41.649596 12959 raft_consensus.cc:482] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:41.649627 12959 raft_consensus.cc:3037] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.650336 12959 raft_consensus.cc:504] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.650475 12959 leader_election.cc:288] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [CANDIDATE]: Term 1 election: Requested vote from peers b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767), 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283)
I20260923 22:17:41.650609 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e80ef250a5b41d2bd39284ec9bf0197" candidate_uuid: "d0aee12b8e784b7c843b24e35062716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f8c19a146643518dcc6e64519b7f7f"
I20260923 22:17:41.650679 12873 raft_consensus.cc:3037] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.650626 12737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e80ef250a5b41d2bd39284ec9bf0197" candidate_uuid: "d0aee12b8e784b7c843b24e35062716a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b453b4f3a8694760bff9c14f49956eab"
I20260923 22:17:41.650729 12737 raft_consensus.cc:3037] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:41.651320 12873 raft_consensus.cc:2454] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aee12b8e784b7c843b24e35062716a in term 1.
I20260923 22:17:41.651489 12531 leader_election.cc:302] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f8c19a146643518dcc6e64519b7f7f, d0aee12b8e784b7c843b24e35062716a; no voters:
I20260923 22:17:41.651546 12737 raft_consensus.cc:2454] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0aee12b8e784b7c843b24e35062716a in term 1.
I20260923 22:17:41.651583 12959 raft_consensus.cc:2781] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:41.651639 12959 raft_consensus.cc:686] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [term 1 LEADER]: Becoming Leader. State: Replica: d0aee12b8e784b7c843b24e35062716a, State: Running, Role: LEADER
I20260923 22:17:41.651691 12959 consensus_queue.cc:237] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.651942 12969 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: TxnStatusTablet state changed. Reason: New leader d0aee12b8e784b7c843b24e35062716a. Latest consensus state: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.652032 12969 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:41.652226 12996 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 22:17:41.652269 12439 catalog_manager.cc:5495] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a reported cstate change: term changed from 0 to 1, leader changed from <none> to d0aee12b8e784b7c843b24e35062716a (127.4.207.1). New cstate: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:41.652356 12996 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 22:17:41.737670 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.735235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.4.207.1:46735)
0923 22:17:41.735291 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.735293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.735303 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:41.735376 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.735377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.735409 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.735498 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:41.735617 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.736179 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.736943 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.737148 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.737230 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.737403 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:41.737439 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:41.737503 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.737569 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.737589 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:17:41.737699 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.735173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:46735 (local address 127.0.0.1:36272)
0923 22:17:41.735252 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.735263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.735312 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.735505 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.735506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.735507 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:41.735559 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.735562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.736257 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.736258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.736843 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.736845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.737227 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.737228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.737291 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.737314 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:41.737503 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:41.737513 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.737565 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.737612 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 22:17:41.738808 12737 raft_consensus.cc:1261] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Refusing update from remote peer d0aee12b8e784b7c843b24e35062716a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:41.738817 12873 raft_consensus.cc:1261] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Refusing update from remote peer d0aee12b8e784b7c843b24e35062716a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:41.739075 12969 consensus_queue.cc:1035] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:41.739198 12959 consensus_queue.cc:1035] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:41.739681 12993 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: TxnStatusTablet state changed. Reason: New leader d0aee12b8e784b7c843b24e35062716a. Latest consensus state: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.739773 12993 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:41.739960 12958 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: TxnStatusTablet state changed. Reason: New leader d0aee12b8e784b7c843b24e35062716a. Latest consensus state: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.740080 12958 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:41.740515 12969 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.740599 12969 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:41.740684 12965 mvcc.cc:205] Tried to move back new op lower bound from 7332666825680330752 to 7332666825325654016. Current Snapshot: MvccSnapshot[applied={T|T < 7332666825680330752}]
I20260923 22:17:41.740790 12993 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.740854 12993 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P 26f8c19a146643518dcc6e64519b7f7f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:41.742130 12959 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.742228 12959 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P d0aee12b8e784b7c843b24e35062716a: This TxnStatusTablet replica's current role is: LEADER
I20260923 22:17:41.742781 12958 tablet_replica.cc:439] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0aee12b8e784b7c843b24e35062716a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } } }
I20260923 22:17:41.742875 12958 tablet_replica.cc:442] T 4e80ef250a5b41d2bd39284ec9bf0197 P b453b4f3a8694760bff9c14f49956eab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 22:17:41.745378 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.742311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.4.207.3:33283)
0923 22:17:41.742596 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.742598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.742606 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:41.742651 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.742651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.742684 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.742790 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:41.743048 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.743751 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.744509 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.744736 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.744806 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.744973 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:41.745086 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:41.745167 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.745275 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.745295 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":62,"threads_started":1}
I20260923 22:17:41.750617 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.747880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:34555 (local address 127.4.207.1:46735)
0923 22:17:41.747933 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.747935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.748040 (+ 105us) server_negotiation.cc:407] Connection header received
0923 22:17:41.748102 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.748103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.748113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.748166 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:41.748281 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.748723 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.749827 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.750092 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.750148 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.750367 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:41.750392 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:41.750427 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.750530 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.750554 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 22:17:41.750653 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.747795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:46735 (local address 127.4.207.3:34555)
0923 22:17:41.747975 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.747986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.748045 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.748175 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.748176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.748178 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:41.748224 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.748225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.748798 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.748799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.749729 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.749732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.750180 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.750182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.750264 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.750280 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:41.750448 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:41.750451 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.750525 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.750570 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
I20260923 22:17:41.759781 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.757243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:39689 (local address 127.4.207.3:33283)
0923 22:17:41.757286 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.757288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.757421 (+ 133us) server_negotiation.cc:407] Connection header received
0923 22:17:41.757470 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.757471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.757480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.757527 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:41.757640 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.758084 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.759181 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.759362 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.759416 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.759594 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:41.759619 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:41.759653 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.759710 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.759728 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:41.759829 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.757218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33283 (local address 127.4.207.1:39689)
0923 22:17:41.757349 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:41.757360 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:41.757416 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:41.757529 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:41.757530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:41.757531 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:41.757586 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.757587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.758156 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.758157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.759104 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:41.759106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.759433 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.759449 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:41.759504 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.759519 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:41.759662 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:41.759663 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 22:17:41.759708 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 22:17:41.759749 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260923 22:17:41.768826 12440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:41.768930 12440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:41.769622 12440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: Sending DeleteTablet for 3 replicas of tablet 114611b3577045a29296466f0acf1f33
I20260923 22:17:41.769902 12857 tablet_service.cc:1514] Processing DeleteTablet for tablet 114611b3577045a29296466f0acf1f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:41 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:53290
I20260923 22:17:41.769917 12717 tablet_service.cc:1514] Processing DeleteTablet for tablet 114611b3577045a29296466f0acf1f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:41 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:59306
I20260923 22:17:41.770092 12577 tablet_service.cc:1514] Processing DeleteTablet for tablet 114611b3577045a29296466f0acf1f33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:41 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:40450
I20260923 22:17:41.770355 13016 tablet_replica.cc:330] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: stopping tablet replica
I20260923 22:17:41.770459 13016 raft_consensus.cc:2229] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:41.773027 13016 raft_consensus.cc:2258] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:41.773368 13016 ts_tablet_manager.cc:1905] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:41.775208 13015 tablet_replica.cc:330] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: stopping tablet replica
I20260923 22:17:41.775337 13015 raft_consensus.cc:2229] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:41.775367 13016 ts_tablet_manager.cc:1918] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:41.775465 13016 log.cc:1196] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/114611b3577045a29296466f0acf1f33
I20260923 22:17:41.775599 13015 raft_consensus.cc:2258] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:41.775720 13016 ts_tablet_manager.cc:1939] T 114611b3577045a29296466f0acf1f33 P b453b4f3a8694760bff9c14f49956eab: Deleting consensus metadata
I20260923 22:17:41.776057 12425 catalog_manager.cc:4841] TS b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767): tablet 114611b3577045a29296466f0acf1f33 (table test-table [id=7b97a03cba0542de90ced95b4ea2d8d1]) successfully deleted
I20260923 22:17:41.776242 13015 ts_tablet_manager.cc:1905] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:41.777908 13015 ts_tablet_manager.cc:1918] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:41.777976 13015 log.cc:1196] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/114611b3577045a29296466f0acf1f33
I20260923 22:17:41.778204 13015 ts_tablet_manager.cc:1939] T 114611b3577045a29296466f0acf1f33 P 26f8c19a146643518dcc6e64519b7f7f: Deleting consensus metadata
I20260923 22:17:41.778200 13017 tablet_replica.cc:330] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: stopping tablet replica
I20260923 22:17:41.778270 13017 raft_consensus.cc:2229] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:41.778355 13017 raft_consensus.cc:2258] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:41.778573 12427 catalog_manager.cc:4841] TS 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283): tablet 114611b3577045a29296466f0acf1f33 (table test-table [id=7b97a03cba0542de90ced95b4ea2d8d1]) successfully deleted
I20260923 22:17:41.778636 13017 ts_tablet_manager.cc:1905] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:41.780666 13017 ts_tablet_manager.cc:1918] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:41.780740 13017 log.cc:1196] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/114611b3577045a29296466f0acf1f33
I20260923 22:17:41.780997 13017 ts_tablet_manager.cc:1939] T 114611b3577045a29296466f0acf1f33 P d0aee12b8e784b7c843b24e35062716a: Deleting consensus metadata
I20260923 22:17:41.781378 12427 catalog_manager.cc:4841] TS d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735): tablet 114611b3577045a29296466f0acf1f33 (table test-table [id=7b97a03cba0542de90ced95b4ea2d8d1]) successfully deleted
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 22:17:41.802515 13025 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:41.795945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.4.207.62:38819)
0923 22:17:41.796055 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.796057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.796124 (+ 67us) server_negotiation.cc:407] Connection header received
0923 22:17:41.796227 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.796227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.796238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.796288 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.797064 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.797603 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.798751 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.798948 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.799004 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.802312 (+ 3308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47508: BlockingRecv error: recv got EOF from 127.0.0.1:47508 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":60,"threads_started":1}
Sep 23 22:17:41 dist-test-slave-7511 krb5kdc[12395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM
I20260923 22:17:41.810883 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:41.803218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.4.207.62:38819)
0923 22:17:41.803265 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:41.803267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:41.803366 (+ 99us) server_negotiation.cc:407] Connection header received
0923 22:17:41.803415 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:41.803416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:41.803429 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:41.803548 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:41.804020 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.804772 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.805463 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:41.805752 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:41.805826 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:41.808164 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:41.808166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:41.808166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:41.808175 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:41.809787 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.810109 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.810111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.810111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.810181 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:41.810355 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:41.810356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:41.810356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:41.810499 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:41.810611 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:41.810690 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 22:17:41.810780 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:17:41.812376 12440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:41.812562 12440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:41.813922 12717 tablet_service.cc:1467] Processing CreateTablet for tablet 419aef9c33fa416ba6faa780ba736a0f (DEFAULT_TABLE table=test-table [id=530a65b618d945ea8fa5abbbff21eb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.814062 12717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419aef9c33fa416ba6faa780ba736a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.814041 12577 tablet_service.cc:1467] Processing CreateTablet for tablet 419aef9c33fa416ba6faa780ba736a0f (DEFAULT_TABLE table=test-table [id=530a65b618d945ea8fa5abbbff21eb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.814185 12577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419aef9c33fa416ba6faa780ba736a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.815676 12857 tablet_service.cc:1467] Processing CreateTablet for tablet 419aef9c33fa416ba6faa780ba736a0f (DEFAULT_TABLE table=test-table [id=530a65b618d945ea8fa5abbbff21eb56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:41.815810 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 419aef9c33fa416ba6faa780ba736a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:41.816118 13031 tablet_bootstrap.cc:492] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: Bootstrap starting.
I20260923 22:17:41.816591 13031 tablet_bootstrap.cc:654] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.817221 13034 tablet_bootstrap.cc:492] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: Bootstrap starting.
I20260923 22:17:41.817306 13031 tablet_bootstrap.cc:492] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: No bootstrap required, opened a new log
I20260923 22:17:41.817358 13031 ts_tablet_manager.cc:1397] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:41.817526 13031 raft_consensus.cc:348] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.817607 13031 raft_consensus.cc:374] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.817634 13031 raft_consensus.cc:729] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b453b4f3a8694760bff9c14f49956eab, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.817728 13034 tablet_bootstrap.cc:654] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.817824 13031 consensus_queue.cc:260] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.818179 13031 ts_tablet_manager.cc:1428] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:41.818300 13032 tablet_bootstrap.cc:492] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: Bootstrap starting.
I20260923 22:17:41.818532 13034 tablet_bootstrap.cc:492] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: No bootstrap required, opened a new log
I20260923 22:17:41.818593 13034 ts_tablet_manager.cc:1397] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:41.818651 13032 tablet_bootstrap.cc:654] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:41.818751 13034 raft_consensus.cc:348] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.818836 13034 raft_consensus.cc:374] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.818861 13034 raft_consensus.cc:729] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f8c19a146643518dcc6e64519b7f7f, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.818914 13034 consensus_queue.cc:260] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.819016 13034 ts_tablet_manager.cc:1428] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:41.819309 13032 tablet_bootstrap.cc:492] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: No bootstrap required, opened a new log
I20260923 22:17:41.819371 13032 ts_tablet_manager.cc:1397] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:41.819514 13032 raft_consensus.cc:348] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.819590 13032 raft_consensus.cc:374] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:41.819614 13032 raft_consensus.cc:729] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aee12b8e784b7c843b24e35062716a, State: Initialized, Role: FOLLOWER
I20260923 22:17:41.819679 13032 consensus_queue.cc:260] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:41.819785 13032 ts_tablet_manager.cc:1428] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:42.017699 12958 raft_consensus.cc:482] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:42.017838 12958 raft_consensus.cc:504] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.018184 12958 leader_election.cc:288] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735), 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283)
I20260923 22:17:42.021365 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.018499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:46735 (local address 127.4.207.2:51453)
0923 22:17:42.018646 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:42.018658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:42.018762 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:42.018972 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:42.018973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:42.018975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:42.019043 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.019045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.019785 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.019786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.020577 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.020579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.021116 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.021118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.021185 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.021190 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:17:42.021243 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:17:42.021279 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":14,"thread_start_us":84,"threads_started":1}
I20260923 22:17:42.021391 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.018579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:51453 (local address 127.4.207.1:46735)
0923 22:17:42.018659 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:42.018661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:42.018749 (+ 88us) server_negotiation.cc:407] Connection header received
0923 22:17:42.018889 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:42.018891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:42.018903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:42.018961 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:42.019117 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.019701 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.020664 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.021032 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.021085 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.021094 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:42.021265 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 22:17:42.021304 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 22:17:42.021502 13038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.018713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33283 (local address 127.4.207.2:37945)
0923 22:17:42.018813 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:42.018828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:42.018907 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:42.019054 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:42.019055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:42.019056 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:17:42.019130 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.019132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.019850 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.019850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.020590 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.020592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.021202 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.021203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.021300 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.021304 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:17:42.021365 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:17:42.021399 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260923 22:17:42.021531 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.018789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:37945 (local address 127.4.207.3:33283)
0923 22:17:42.018835 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:42.018837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:42.018909 (+ 72us) server_negotiation.cc:407] Connection header received
0923 22:17:42.018976 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:42.018977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:42.018986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:42.019045 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:17:42.019198 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.019744 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.020696 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.021128 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.021194 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.021206 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:42.021380 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 22:17:42.021433 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:17:42.021723 12597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419aef9c33fa416ba6faa780ba736a0f" candidate_uuid: "b453b4f3a8694760bff9c14f49956eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aee12b8e784b7c843b24e35062716a" is_pre_election: true
I20260923 22:17:42.021833 12597 raft_consensus.cc:2454] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b453b4f3a8694760bff9c14f49956eab in term 0.
I20260923 22:17:42.021793 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419aef9c33fa416ba6faa780ba736a0f" candidate_uuid: "b453b4f3a8694760bff9c14f49956eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f8c19a146643518dcc6e64519b7f7f" is_pre_election: true
I20260923 22:17:42.021891 12873 raft_consensus.cc:2454] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b453b4f3a8694760bff9c14f49956eab in term 0.
I20260923 22:17:42.022027 12671 leader_election.cc:302] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b453b4f3a8694760bff9c14f49956eab, d0aee12b8e784b7c843b24e35062716a; no voters:
I20260923 22:17:42.022157 12958 raft_consensus.cc:2781] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:42.022197 12958 raft_consensus.cc:482] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:42.022224 12958 raft_consensus.cc:3037] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:42.022876 12958 raft_consensus.cc:504] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.023015 12958 leader_election.cc:288] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [CANDIDATE]: Term 1 election: Requested vote from peers d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735), 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283)
I20260923 22:17:42.023178 12597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419aef9c33fa416ba6faa780ba736a0f" candidate_uuid: "b453b4f3a8694760bff9c14f49956eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aee12b8e784b7c843b24e35062716a"
I20260923 22:17:42.023193 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "419aef9c33fa416ba6faa780ba736a0f" candidate_uuid: "b453b4f3a8694760bff9c14f49956eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f8c19a146643518dcc6e64519b7f7f"
I20260923 22:17:42.023265 12597 raft_consensus.cc:3037] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:42.023265 12873 raft_consensus.cc:3037] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:42.024094 12597 raft_consensus.cc:2454] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b453b4f3a8694760bff9c14f49956eab in term 1.
I20260923 22:17:42.024094 12873 raft_consensus.cc:2454] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b453b4f3a8694760bff9c14f49956eab in term 1.
I20260923 22:17:42.024257 12671 leader_election.cc:302] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f8c19a146643518dcc6e64519b7f7f, b453b4f3a8694760bff9c14f49956eab; no voters:
I20260923 22:17:42.024343 12958 raft_consensus.cc:2781] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:42.024473 12958 raft_consensus.cc:686] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 1 LEADER]: Becoming Leader. State: Replica: b453b4f3a8694760bff9c14f49956eab, State: Running, Role: LEADER
I20260923 22:17:42.024549 12958 consensus_queue.cc:237] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.025288 12440 catalog_manager.cc:5495] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab reported cstate change: term changed from 0 to 1, leader changed from <none> to b453b4f3a8694760bff9c14f49956eab (127.4.207.2). New cstate: current_term: 1 leader_uuid: "b453b4f3a8694760bff9c14f49956eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:42.046058 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.043425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.4.207.2:34767)
0923 22:17:42.043614 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:42.043615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:42.043624 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:42.043666 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:42.043666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:42.043687 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:42.043776 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:42.043935 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.044550 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.045375 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.045571 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.045623 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.045760 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:42.045869 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:42.045926 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:42.045960 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 22:17:42.045977 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":123,"threads_started":1}
I20260923 22:17:42.047114 12597 raft_consensus.cc:1261] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Refusing update from remote peer b453b4f3a8694760bff9c14f49956eab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:42.047114 12873 raft_consensus.cc:1261] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Refusing update from remote peer b453b4f3a8694760bff9c14f49956eab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:42.047426 13039 consensus_queue.cc:1035] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:42.047585 12958 consensus_queue.cc:1035] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:42.050908 12440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:42.051017 12440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:42.051683 12440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: Sending DeleteTablet for 3 replicas of tablet 419aef9c33fa416ba6faa780ba736a0f
I20260923 22:17:42.051945 12717 tablet_service.cc:1514] Processing DeleteTablet for tablet 419aef9c33fa416ba6faa780ba736a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:42 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:59306
I20260923 22:17:42.051980 12577 tablet_service.cc:1514] Processing DeleteTablet for tablet 419aef9c33fa416ba6faa780ba736a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:42 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:40450
I20260923 22:17:42.052084 13016 tablet_replica.cc:330] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: stopping tablet replica
I20260923 22:17:42.052148 13016 raft_consensus.cc:2229] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:42.052227 13016 raft_consensus.cc:2258] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:42.052311 13017 tablet_replica.cc:330] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: stopping tablet replica
I20260923 22:17:42.052371 13017 raft_consensus.cc:2229] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:42.052419 13017 raft_consensus.cc:2258] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:42.052531 13016 ts_tablet_manager.cc:1905] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:42.052610 12857 tablet_service.cc:1514] Processing DeleteTablet for tablet 419aef9c33fa416ba6faa780ba736a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:42 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:53290
I20260923 22:17:42.052698 13017 ts_tablet_manager.cc:1905] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:42.052719 13015 tablet_replica.cc:330] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: stopping tablet replica
I20260923 22:17:42.052767 13015 raft_consensus.cc:2229] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:42.052812 13015 raft_consensus.cc:2258] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:42.052968 12440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:17:42.053105 12440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:17:42.054277 13016 ts_tablet_manager.cc:1918] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:42.054288 12577 tablet_service.cc:1467] Processing CreateTablet for tablet aa920a6762534910aa27b154fce75a39 (DEFAULT_TABLE table=test-table [id=d4c0a88c3b1b4aefb4fa3f7d7b10d1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:42.054352 13016 log.cc:1196] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/419aef9c33fa416ba6faa780ba736a0f
I20260923 22:17:42.054363 13017 ts_tablet_manager.cc:1918] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:42.054389 12577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa920a6762534910aa27b154fce75a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:42.054409 13017 log.cc:1196] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/419aef9c33fa416ba6faa780ba736a0f
I20260923 22:17:42.054579 13016 ts_tablet_manager.cc:1939] T 419aef9c33fa416ba6faa780ba736a0f P b453b4f3a8694760bff9c14f49956eab: Deleting consensus metadata
I20260923 22:17:42.054626 13017 ts_tablet_manager.cc:1939] T 419aef9c33fa416ba6faa780ba736a0f P d0aee12b8e784b7c843b24e35062716a: Deleting consensus metadata
I20260923 22:17:42.055629 13032 tablet_bootstrap.cc:492] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: Bootstrap starting.
I20260923 22:17:42.055680 12425 catalog_manager.cc:4841] TS b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767): tablet 419aef9c33fa416ba6faa780ba736a0f (table test-table [id=530a65b618d945ea8fa5abbbff21eb56]) successfully deleted
I20260923 22:17:42.055881 12717 tablet_service.cc:1467] Processing CreateTablet for tablet aa920a6762534910aa27b154fce75a39 (DEFAULT_TABLE table=test-table [id=d4c0a88c3b1b4aefb4fa3f7d7b10d1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:42.055927 13032 tablet_bootstrap.cc:654] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:42.055976 12717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa920a6762534910aa27b154fce75a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:42.056977 13031 tablet_bootstrap.cc:492] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: Bootstrap starting.
I20260923 22:17:42.057000 12857 tablet_service.cc:1467] Processing CreateTablet for tablet aa920a6762534910aa27b154fce75a39 (DEFAULT_TABLE table=test-table [id=d4c0a88c3b1b4aefb4fa3f7d7b10d1dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:17:42.057128 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa920a6762534910aa27b154fce75a39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:42.057430 13031 tablet_bootstrap.cc:654] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:42.059633 13015 ts_tablet_manager.cc:1905] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:42.061427 13015 ts_tablet_manager.cc:1918] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:17:42.061489 13015 log.cc:1196] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/419aef9c33fa416ba6faa780ba736a0f
I20260923 22:17:42.061715 13015 ts_tablet_manager.cc:1939] T 419aef9c33fa416ba6faa780ba736a0f P 26f8c19a146643518dcc6e64519b7f7f: Deleting consensus metadata
I20260923 22:17:42.062121 12427 catalog_manager.cc:4841] TS 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283): tablet 419aef9c33fa416ba6faa780ba736a0f (table test-table [id=530a65b618d945ea8fa5abbbff21eb56]) successfully deleted
I20260923 22:17:42.062322 12427 catalog_manager.cc:4841] TS d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735): tablet 419aef9c33fa416ba6faa780ba736a0f (table test-table [id=530a65b618d945ea8fa5abbbff21eb56]) successfully deleted
I20260923 22:17:42.062745 13034 tablet_bootstrap.cc:492] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: Bootstrap starting.
I20260923 22:17:42.063104 13034 tablet_bootstrap.cc:654] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:42.063975 13032 tablet_bootstrap.cc:492] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: No bootstrap required, opened a new log
I20260923 22:17:42.064046 13032 ts_tablet_manager.cc:1397] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260923 22:17:42.064218 13032 raft_consensus.cc:348] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.064307 13032 raft_consensus.cc:374] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:42.064332 13032 raft_consensus.cc:729] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0aee12b8e784b7c843b24e35062716a, State: Initialized, Role: FOLLOWER
I20260923 22:17:42.064386 13032 consensus_queue.cc:260] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.064522 13032 ts_tablet_manager.cc:1428] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:42.065032 13034 tablet_bootstrap.cc:492] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: No bootstrap required, opened a new log
I20260923 22:17:42.065104 13034 ts_tablet_manager.cc:1397] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:17:42.065124 13031 tablet_bootstrap.cc:492] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: No bootstrap required, opened a new log
I20260923 22:17:42.065177 13031 ts_tablet_manager.cc:1397] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260923 22:17:42.065222 13034 raft_consensus.cc:348] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.065307 13034 raft_consensus.cc:374] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:42.065327 13034 raft_consensus.cc:729] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f8c19a146643518dcc6e64519b7f7f, State: Initialized, Role: FOLLOWER
I20260923 22:17:42.065371 13034 consensus_queue.cc:260] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.065330 13031 raft_consensus.cc:348] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.065419 13031 raft_consensus.cc:374] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:42.065451 13031 raft_consensus.cc:729] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b453b4f3a8694760bff9c14f49956eab, State: Initialized, Role: FOLLOWER
I20260923 22:17:42.065464 13034 ts_tablet_manager.cc:1428] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:42.065491 13031 consensus_queue.cc:260] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.065596 13031 ts_tablet_manager.cc:1428] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:42.142992 13045 raft_consensus.cc:482] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:17:42.143100 13045 raft_consensus.cc:504] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.143270 13045 leader_election.cc:288] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735), 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283)
I20260923 22:17:42.143522 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa920a6762534910aa27b154fce75a39" candidate_uuid: "b453b4f3a8694760bff9c14f49956eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f8c19a146643518dcc6e64519b7f7f" is_pre_election: true
I20260923 22:17:42.143617 12873 raft_consensus.cc:2454] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b453b4f3a8694760bff9c14f49956eab in term 0.
I20260923 22:17:42.143587 12597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa920a6762534910aa27b154fce75a39" candidate_uuid: "b453b4f3a8694760bff9c14f49956eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aee12b8e784b7c843b24e35062716a" is_pre_election: true
I20260923 22:17:42.143698 12597 raft_consensus.cc:2454] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b453b4f3a8694760bff9c14f49956eab in term 0.
I20260923 22:17:42.143792 12671 leader_election.cc:302] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f8c19a146643518dcc6e64519b7f7f, b453b4f3a8694760bff9c14f49956eab; no voters:
I20260923 22:17:42.143870 13045 raft_consensus.cc:2781] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:17:42.143919 13045 raft_consensus.cc:482] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:17:42.143947 13045 raft_consensus.cc:3037] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:42.144891 13045 raft_consensus.cc:504] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.145030 13045 leader_election.cc:288] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [CANDIDATE]: Term 1 election: Requested vote from peers d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735), 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283)
I20260923 22:17:42.145228 12873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa920a6762534910aa27b154fce75a39" candidate_uuid: "b453b4f3a8694760bff9c14f49956eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f8c19a146643518dcc6e64519b7f7f"
I20260923 22:17:42.145241 12597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa920a6762534910aa27b154fce75a39" candidate_uuid: "b453b4f3a8694760bff9c14f49956eab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0aee12b8e784b7c843b24e35062716a"
I20260923 22:17:42.145313 12873 raft_consensus.cc:3037] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:42.145327 12597 raft_consensus.cc:3037] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:42.146004 12597 raft_consensus.cc:2454] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b453b4f3a8694760bff9c14f49956eab in term 1.
I20260923 22:17:42.146145 12873 raft_consensus.cc:2454] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b453b4f3a8694760bff9c14f49956eab in term 1.
I20260923 22:17:42.146169 12671 leader_election.cc:302] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b453b4f3a8694760bff9c14f49956eab, d0aee12b8e784b7c843b24e35062716a; no voters:
I20260923 22:17:42.146281 13045 raft_consensus.cc:2781] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:42.146327 13045 raft_consensus.cc:686] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 1 LEADER]: Becoming Leader. State: Replica: b453b4f3a8694760bff9c14f49956eab, State: Running, Role: LEADER
I20260923 22:17:42.146384 13045 consensus_queue.cc:237] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } }
I20260923 22:17:42.147051 12439 catalog_manager.cc:5495] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab reported cstate change: term changed from 0 to 1, leader changed from <none> to b453b4f3a8694760bff9c14f49956eab (127.4.207.2). New cstate: current_term: 1 leader_uuid: "b453b4f3a8694760bff9c14f49956eab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b453b4f3a8694760bff9c14f49956eab" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 34767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 } health_report { overall_health: UNKNOWN } } }
I20260923 22:17:42.168915 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.166191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:34343 (local address 127.4.207.1:46735)
0923 22:17:42.166242 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:42.166244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:42.166384 (+ 140us) server_negotiation.cc:407] Connection header received
0923 22:17:42.166445 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:42.166445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:42.166454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:42.166496 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:42.166642 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.167113 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.168248 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.168436 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.168492 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.168688 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:42.168716 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:42.168753 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:42.168831 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 22:17:42.168854 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:17:42.168975 13053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.166113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:46735 (local address 127.4.207.2:34343)
0923 22:17:42.166303 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:42.166316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:42.166383 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:42.166509 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:42.166510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:42.166512 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:42.166582 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.166584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.167194 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.167195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.168170 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.168173 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.168528 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.168529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.168593 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.168608 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:42.168772 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:42.168774 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:17:42.168829 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:17:42.168874 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":107,"threads_started":1}
I20260923 22:17:42.171058 12597 raft_consensus.cc:1261] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Refusing update from remote peer b453b4f3a8694760bff9c14f49956eab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:42.171090 12873 raft_consensus.cc:1261] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Refusing update from remote peer b453b4f3a8694760bff9c14f49956eab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:17:42.171277 13045 consensus_queue.cc:1035] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f8c19a146643518dcc6e64519b7f7f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:42.171777 13045 consensus_queue.cc:1035] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0aee12b8e784b7c843b24e35062716a" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 46735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:17:42.178539 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.175808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:41297 (local address 127.4.207.2:34767)
0923 22:17:42.175846 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:42.175848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:42.175857 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:42.175881 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:42.175882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:42.175893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:42.175945 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:17:42.176077 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.176818 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.177829 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.178041 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.178105 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.178252 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:17:42.178287 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:17:42.178348 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:42.178425 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 22:17:42.178450 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:17:42.178551 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:42.175718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:34767 (local address 127.4.207.1:41297)
0923 22:17:42.175762 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:42.175774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:42.175811 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:42.175960 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:42.175961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:42.175963 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 22:17:42.176009 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.176011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.176881 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.176882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.177747 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:42.177750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:42.178102 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:42.178103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:42.178161 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:42.178167 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 22:17:42.178351 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 22:17:42.178354 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:17:42.178419 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 22:17:42.178473 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 22:17:42.184430 12439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:42.184572 12439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:17:42.185266 12439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bd61e55bd44451af58188af50b666f: Sending DeleteTablet for 3 replicas of tablet aa920a6762534910aa27b154fce75a39
I20260923 22:17:42.185545 12577 tablet_service.cc:1514] Processing DeleteTablet for tablet aa920a6762534910aa27b154fce75a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:42 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:40450
I20260923 22:17:42.185540 12857 tablet_service.cc:1514] Processing DeleteTablet for tablet aa920a6762534910aa27b154fce75a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:42 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:53290
I20260923 22:17:42.185541 12717 tablet_service.cc:1514] Processing DeleteTablet for tablet aa920a6762534910aa27b154fce75a39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:17:42 UTC) from {username='oryx', principal='oryx/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:59306
I20260923 22:17:42.185636 13017 tablet_replica.cc:330] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: stopping tablet replica
I20260923 22:17:42.185636 13015 tablet_replica.cc:330] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: stopping tablet replica
I20260923 22:17:42.185640 13016 tablet_replica.cc:330] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: stopping tablet replica
I20260923 22:17:42.185683 13016 raft_consensus.cc:2229] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:17:42.185685 13017 raft_consensus.cc:2229] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:42.185690 13015 raft_consensus.cc:2229] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:17:42.185721 13017 raft_consensus.cc:2258] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:42.185722 13015 raft_consensus.cc:2258] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:42.185742 13016 raft_consensus.cc:2258] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:17:42.186017 13017 ts_tablet_manager.cc:1905] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:42.186381 13016 ts_tablet_manager.cc:1905] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:42.187755 13017 ts_tablet_manager.cc:1918] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:42.187824 13017 log.cc:1196] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/aa920a6762534910aa27b154fce75a39
I20260923 22:17:42.187944 13016 ts_tablet_manager.cc:1918] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:42.188004 13016 log.cc:1196] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/aa920a6762534910aa27b154fce75a39
I20260923 22:17:42.188082 13017 ts_tablet_manager.cc:1939] T aa920a6762534910aa27b154fce75a39 P d0aee12b8e784b7c843b24e35062716a: Deleting consensus metadata
I20260923 22:17:42.188217 13016 ts_tablet_manager.cc:1939] T aa920a6762534910aa27b154fce75a39 P b453b4f3a8694760bff9c14f49956eab: Deleting consensus metadata
I20260923 22:17:42.188419 12427 catalog_manager.cc:4841] TS d0aee12b8e784b7c843b24e35062716a (127.4.207.1:46735): tablet aa920a6762534910aa27b154fce75a39 (table test-table [id=d4c0a88c3b1b4aefb4fa3f7d7b10d1dc]) successfully deleted
I20260923 22:17:42.188624 12425 catalog_manager.cc:4841] TS b453b4f3a8694760bff9c14f49956eab (127.4.207.2:34767): tablet aa920a6762534910aa27b154fce75a39 (table test-table [id=d4c0a88c3b1b4aefb4fa3f7d7b10d1dc]) successfully deleted
I20260923 22:17:42.188761 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 12512
I20260923 22:17:42.195086 13015 ts_tablet_manager.cc:1905] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:17:42.196764 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 12652
I20260923 22:17:42.197021 13015 ts_tablet_manager.cc:1918] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 22:17:42.197086 13015 log.cc:1196] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/aa920a6762534910aa27b154fce75a39
I20260923 22:17:42.197274 13015 ts_tablet_manager.cc:1939] T aa920a6762534910aa27b154fce75a39 P 26f8c19a146643518dcc6e64519b7f7f: Deleting consensus metadata
I20260923 22:17:42.198449 12427 catalog_manager.cc:4841] TS 26f8c19a146643518dcc6e64519b7f7f (127.4.207.3:33283): tablet aa920a6762534910aa27b154fce75a39 (table test-table [id=d4c0a88c3b1b4aefb4fa3f7d7b10d1dc]) successfully deleted
I20260923 22:17:42.202775 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 12792
I20260923 22:17:42.208474 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 12408
2026-09-23T22:17:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2864 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:42 dist-test-slave-7511 krb5kdc[13059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:42 dist-test-slave-7511 krb5kdc[13059](info): set up 2 sockets
Sep 23 22:17:42 dist-test-slave-7511 krb5kdc[13059](info): commencing operation
krb5kdc: starting...
W20260923 22:17:43.244252 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:43 dist-test-slave-7511 krb5kdc[13059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:43.364127 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42299
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:42299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:43.447077 13072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:43.447320 13072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:43.447345 13072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:43.448700 13072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:43.448747 13072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:43.448762 13072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:43.448776 13072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:43.448788 13072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:43.450392 13072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:42299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:43.450655 13072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:43.450907 13072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:43.456106 13078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:43.456135 13072 server_base.cc:1034] running on GCE node
W20260923 22:17:43.456102 13080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:43.456102 13077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:43.456696 13072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:43.457026 13072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:43.458197 13072 hybrid_clock.cc:648] HybridClock initialized: now 1790201863458193 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T22:17:43Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1252 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:17:43 dist-test-slave-7511 krb5kdc[13085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:17:43 dist-test-slave-7511 krb5kdc[13085](info): set up 2 sockets
Sep 23 22:17:43 dist-test-slave-7511 krb5kdc[13085](info): commencing operation
krb5kdc: starting...
W20260923 22:17:44.500119 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:17:44 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:17:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:17:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:17:44.622822 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35521
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:44.705380 13098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:44.705642 13098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:44.705682 13098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:44.706885 13098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:44.706955 13098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:44.706979 13098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:44.707000 13098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:44.707028 13098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:44.708591 13098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:44.708874 13098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:44.709138 13098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:44.715219 13103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:44.715272 13106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:44.715273 13104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:44.715293 13098 server_base.cc:1034] running on GCE node
I20260923 22:17:44.715726 13098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:44.716080 13098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:44.717245 13098 hybrid_clock.cc:648] HybridClock initialized: now 1790201864717225 us; error 35 us; skew 500 ppm
Sep 23 22:17:44 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201864, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:44.719714 13098 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:44.720211 13098 webserver.cc:460] Webserver started at http://127.4.207.62:36621/ using document root <none> and password file <none>
I20260923 22:17:44.720469 13098 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:44.720537 13098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:44.720675 13098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:44.721777 13098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "af241fb6e3a74c0191f0004360b9550a"
format_stamp: "Formatted at 2026-09-23 22:17:44 on dist-test-slave-7511"
server_key: "ccbaa7b17ff5fe1f922386c398cff86c"
server_key_iv: "1cafcca1ad95becfc39730833b80d884"
server_key_version: "encryptionkey@0"
I20260923 22:17:44.722136 13098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "af241fb6e3a74c0191f0004360b9550a"
format_stamp: "Formatted at 2026-09-23 22:17:44 on dist-test-slave-7511"
server_key: "ccbaa7b17ff5fe1f922386c398cff86c"
server_key_iv: "1cafcca1ad95becfc39730833b80d884"
server_key_version: "encryptionkey@0"
I20260923 22:17:44.723537 13098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:17:44.724262 13113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:44.724426 13098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:44.724526 13098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "af241fb6e3a74c0191f0004360b9550a"
format_stamp: "Formatted at 2026-09-23 22:17:44 on dist-test-slave-7511"
server_key: "ccbaa7b17ff5fe1f922386c398cff86c"
server_key_iv: "1cafcca1ad95becfc39730833b80d884"
server_key_version: "encryptionkey@0"
I20260923 22:17:44.724606 13098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:44.734779 13098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:44.736822 13098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:44.736984 13098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:44.741076 13098 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:35521
I20260923 22:17:44.741110 13165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:35521 every 8 connection(s)
I20260923 22:17:44.741492 13098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:44.742336 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:17:44.744729 13166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Bootstrap starting.
I20260923 22:17:44.745329 13166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Neither blocks nor log segments found. Creating new log.
I20260923 22:17:44.745618 13166 log.cc:824] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:44.746322 13166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: No bootstrap required, opened a new log
I20260923 22:17:44.747594 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 13098
I20260923 22:17:44.747704 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:17:44.747891 4924 external_mini_cluster.cc:1428] Setting key e6908d9b55dfd435b809ace9b2e5d246
I20260923 22:17:44.748303 13166 raft_consensus.cc:348] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } }
I20260923 22:17:44.748503 13166 raft_consensus.cc:374] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:17:44.748602 13166 raft_consensus.cc:729] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af241fb6e3a74c0191f0004360b9550a, State: Initialized, Role: FOLLOWER
I20260923 22:17:44.748764 13166 consensus_queue.cc:260] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } }
I20260923 22:17:44.748847 13166 raft_consensus.cc:388] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:44.748878 13166 raft_consensus.cc:482] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:44.748922 13166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:17:44.750113 13166 raft_consensus.cc:504] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } }
I20260923 22:17:44.750204 13166 leader_election.cc:302] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af241fb6e3a74c0191f0004360b9550a; no voters:
I20260923 22:17:44.750375 13166 leader_election.cc:288] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:17:44.750434 13171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:17:44.750588 13171 raft_consensus.cc:686] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 1 LEADER]: Becoming Leader. State: Replica: af241fb6e3a74c0191f0004360b9550a, State: Running, Role: LEADER
I20260923 22:17:44.750645 13166 sys_catalog.cc:564] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:44.750667 13171 consensus_queue.cc:237] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } }
I20260923 22:17:44.750963 13171 sys_catalog.cc:455] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: SysCatalogTable state changed. Reason: New leader af241fb6e3a74c0191f0004360b9550a. Latest consensus state: current_term: 1 leader_uuid: "af241fb6e3a74c0191f0004360b9550a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } } }
I20260923 22:17:44.751031 13171 sys_catalog.cc:458] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:44.751021 13172 sys_catalog.cc:455] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af241fb6e3a74c0191f0004360b9550a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } } }
I20260923 22:17:44.751482 13172 sys_catalog.cc:458] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:44.752003 13179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:44.752571 13179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:44.754272 13179 catalog_manager.cc:1347] Generated new cluster ID: f77f5524424a493399bbfa8b6cddddc8
I20260923 22:17:44.754335 13179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:17:44 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:44.759315 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:44.748447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.4.207.62:35521)
0923 22:17:44.748642 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:44.748653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:44.748687 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:17:44.748814 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:44.748821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:44.748881 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:44.749037 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:44.749985 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.751461 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.752589 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.753190 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.753311 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.755997 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:44.756001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:44.756002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:44.756015 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:44.757696 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:44.758084 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:44.758087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:44.758087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:44.758189 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:44.758393 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:44.758394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:44.758394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:44.758646 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:44.758853 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:44.758922 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 22:17:44.759160 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260923 22:17:44.769582 13179 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:17:44.770141 13179 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:44.782850 13179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Generated new TSK 0
I20260923 22:17:44.783073 13179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:17:44.814368 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35521
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:44.923301 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:44.923545 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:44.923571 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:44.924753 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:44.924796 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:44.924825 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:44.926283 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:44.926489 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:44.926690 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:44.931958 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:44.931949 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:44.931949 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:44.932174 13192 server_base.cc:1034] running on GCE node
I20260923 22:17:44.932399 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:44.932603 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:44.933780 13192 hybrid_clock.cc:648] HybridClock initialized: now 1790201864933769 us; error 27 us; skew 500 ppm
Sep 23 22:17:44 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201864, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:44.936342 13192 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:17:44.936818 13192 webserver.cc:460] Webserver started at http://127.4.207.1:39789/ using document root <none> and password file <none>
I20260923 22:17:44.937042 13192 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:44.937088 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:44.937273 13192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:44.938347 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "56148672e32341d59016b05942e17e12"
format_stamp: "Formatted at 2026-09-23 22:17:44 on dist-test-slave-7511"
server_key: "8dc59e8cdb620475d6aec5f8d5af55a3"
server_key_iv: "096e5a8bd50d7d09421f7afc0f01a0a1"
server_key_version: "encryptionkey@0"
I20260923 22:17:44.938709 13192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "56148672e32341d59016b05942e17e12"
format_stamp: "Formatted at 2026-09-23 22:17:44 on dist-test-slave-7511"
server_key: "8dc59e8cdb620475d6aec5f8d5af55a3"
server_key_iv: "096e5a8bd50d7d09421f7afc0f01a0a1"
server_key_version: "encryptionkey@0"
I20260923 22:17:44.940166 13192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 22:17:44.940876 13207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:44.941033 13192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:44.941121 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "56148672e32341d59016b05942e17e12"
format_stamp: "Formatted at 2026-09-23 22:17:44 on dist-test-slave-7511"
server_key: "8dc59e8cdb620475d6aec5f8d5af55a3"
server_key_iv: "096e5a8bd50d7d09421f7afc0f01a0a1"
server_key_version: "encryptionkey@0"
I20260923 22:17:44.941195 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:44.965117 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:44.967226 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:44.967372 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:44.967669 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:44.968008 13192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:44.968043 13192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:44.968067 13192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:44.968112 13192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:44.973908 13192 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:34617
I20260923 22:17:44.973973 13320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:34617 every 8 connection(s)
I20260923 22:17:44.974292 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:17:44 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201864, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:44.980497 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 13192
I20260923 22:17:44.980623 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:17:44.980814 4924 external_mini_cluster.cc:1428] Setting key a7efb4a6f1482e5ffc84efd2ff857f89
I20260923 22:17:44.984025 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:44.975124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:41187 (local address 127.4.207.62:35521)
0923 22:17:44.975181 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:44.975183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:44.975375 (+ 192us) server_negotiation.cc:407] Connection header received
0923 22:17:44.975864 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:44.975865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:44.975879 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:44.975942 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:44.976879 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.977399 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.978378 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.978575 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.978633 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.980796 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:44.980797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:44.980798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:44.980807 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:44.982411 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:44.982947 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:44.982948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:44.982948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:44.983035 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:44.983285 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:44.983286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:44.983286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:44.983471 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:44.983547 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:44.983801 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 22:17:44.983901 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:17:44.984058 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:44.975084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:41187)
0923 22:17:44.975246 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:44.975290 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:44.975780 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:44.975995 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:44.975999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:44.976580 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:44.976787 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:44.976793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.977482 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.977482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:44.978293 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:44.978295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:44.978649 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:44.978650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:44.978748 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:44.979205 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:44.979222 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:44.980709 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:44.982515 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:44.982522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:44.982524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:44.982871 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:44.983104 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:44.983105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:44.983105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:44.983221 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:44.983536 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:44.983541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:44.983663 (+ 122us) client_negotiation.cc:769] Sending connection context
0923 22:17:44.983762 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 22:17:44.983888 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260923 22:17:44.984593 13321 heartbeater.cc:347] Connected to a master server at 127.4.207.62:35521
I20260923 22:17:44.984710 13321 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:44.984889 13321 heartbeater.cc:511] Master 127.4.207.62:35521 requested a full tablet report, sending...
I20260923 22:17:44.985456 13130 ts_manager.cc:194] Registered new tserver with Master: 56148672e32341d59016b05942e17e12 (127.4.207.1:34617)
I20260923 22:17:44.986060 13130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:41187
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:17:45.006335 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35521
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:45.088406 13326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:45.088657 13326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:45.088696 13326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:45.089903 13326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:45.089977 13326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:45.090022 13326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:45.091558 13326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:45.091842 13326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:45.092083 13326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:45.097347 13332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.097337 13334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.097337 13331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:45.097471 13326 server_base.cc:1034] running on GCE node
I20260923 22:17:45.097826 13326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:45.098089 13326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:45.099229 13326 hybrid_clock.cc:648] HybridClock initialized: now 1790201865099212 us; error 36 us; skew 500 ppm
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201865, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:45.101876 13326 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:17:45.102342 13326 webserver.cc:460] Webserver started at http://127.4.207.2:46579/ using document root <none> and password file <none>
I20260923 22:17:45.102538 13326 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:45.102579 13326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:45.102671 13326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:45.103801 13326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "574ad5fad65c42b48a7b84870a836219"
format_stamp: "Formatted at 2026-09-23 22:17:45 on dist-test-slave-7511"
server_key: "4b48b98f5f92b7813015cf7ad31a1513"
server_key_iv: "7abf33782b46b653eeaa240588c0680c"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.104101 13326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "574ad5fad65c42b48a7b84870a836219"
format_stamp: "Formatted at 2026-09-23 22:17:45 on dist-test-slave-7511"
server_key: "4b48b98f5f92b7813015cf7ad31a1513"
server_key_iv: "7abf33782b46b653eeaa240588c0680c"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.105373 13326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:45.106037 13341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.106206 13326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.106312 13326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "574ad5fad65c42b48a7b84870a836219"
format_stamp: "Formatted at 2026-09-23 22:17:45 on dist-test-slave-7511"
server_key: "4b48b98f5f92b7813015cf7ad31a1513"
server_key_iv: "7abf33782b46b653eeaa240588c0680c"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.106384 13326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:45.125156 13326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:45.127761 13326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:45.127925 13326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:45.128188 13326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:45.128539 13326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:45.128582 13326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.128604 13326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:45.128618 13326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.134378 13326 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:37265
I20260923 22:17:45.134449 13454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:37265 every 8 connection(s)
I20260923 22:17:45.134764 13326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201865, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:45.141355 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 13326
I20260923 22:17:45.141505 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:17:45.141754 4924 external_mini_cluster.cc:1428] Setting key 616293a575b89dab1a3fe550f9303f39
I20260923 22:17:45.144394 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.135531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:52761 (local address 127.4.207.62:35521)
0923 22:17:45.135579 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.135581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.135805 (+ 224us) server_negotiation.cc:407] Connection header received
0923 22:17:45.136304 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.136305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.136315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.136369 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.137275 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.137782 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.138752 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.138942 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.138990 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.141157 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:45.141158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:45.141159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:45.141167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:45.142758 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:45.143218 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:45.143219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:45.143219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:45.143278 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:45.143584 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:45.143585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:45.143585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:45.143753 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:45.143838 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:45.144235 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 22:17:45.144326 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:17:45.144635 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.135549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:52761)
0923 22:17:45.135679 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.135715 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.136219 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.136425 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.136428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.136997 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.137204 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.137208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.137856 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.137856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.138682 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.138684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.139005 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.139005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.139114 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.139658 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.139669 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.141045 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:45.142868 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:45.142876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:45.142878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:45.143115 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:45.143345 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:45.143346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:45.143346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:45.143513 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:45.143872 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:45.143879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:45.144065 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 22:17:45.144177 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 22:17:45.144365 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
I20260923 22:17:45.145169 13455 heartbeater.cc:347] Connected to a master server at 127.4.207.62:35521
I20260923 22:17:45.145284 13455 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:45.145466 13455 heartbeater.cc:511] Master 127.4.207.62:35521 requested a full tablet report, sending...
I20260923 22:17:45.145846 13130 ts_manager.cc:194] Registered new tserver with Master: 574ad5fad65c42b48a7b84870a836219 (127.4.207.2:37265)
I20260923 22:17:45.146224 13130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:52761
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:17:45.167809 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35521
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:45.250455 13460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:45.250697 13460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:45.250741 13460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:45.252274 13460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:45.252345 13460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:45.252394 13460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:45.253916 13460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:45.254215 13460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:45.254462 13460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:45.259565 13468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.259629 13465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:45.259824 13460 server_base.cc:1034] running on GCE node
W20260923 22:17:45.259567 13466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:45.260113 13460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:45.260397 13460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:45.261560 13460 hybrid_clock.cc:648] HybridClock initialized: now 1790201865261554 us; error 37 us; skew 500 ppm
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201865, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:45.264088 13460 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:17:45.264528 13460 webserver.cc:460] Webserver started at http://127.4.207.3:37693/ using document root <none> and password file <none>
I20260923 22:17:45.264739 13460 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:45.264782 13460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:45.264873 13460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:17:45.265938 13460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "bfcedf640b93495186de562b998b5521"
format_stamp: "Formatted at 2026-09-23 22:17:45 on dist-test-slave-7511"
server_key: "5b27492a415be4202fc763f833a2b8fb"
server_key_iv: "22fe340af4558130897877c7baa593c1"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.266255 13460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "bfcedf640b93495186de562b998b5521"
format_stamp: "Formatted at 2026-09-23 22:17:45 on dist-test-slave-7511"
server_key: "5b27492a415be4202fc763f833a2b8fb"
server_key_iv: "22fe340af4558130897877c7baa593c1"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.267582 13460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:17:45.268384 13475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.268596 13460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:45.268695 13460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "bfcedf640b93495186de562b998b5521"
format_stamp: "Formatted at 2026-09-23 22:17:45 on dist-test-slave-7511"
server_key: "5b27492a415be4202fc763f833a2b8fb"
server_key_iv: "22fe340af4558130897877c7baa593c1"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.268754 13460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:45.288198 13460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:45.290237 13460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:45.290361 13460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:45.290555 13460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:45.290894 13460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:45.290928 13460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.290951 13460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:45.290985 13460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.296909 13460 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:45695
I20260923 22:17:45.296963 13588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:45695 every 8 connection(s)
I20260923 22:17:45.297299 13460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:17:45.302646 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 13460
I20260923 22:17:45.302783 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:17:45.302955 4924 external_mini_cluster.cc:1428] Setting key 710d63006b71ce0a05ed49d2198892d1
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201865, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:17:45.306784 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.298062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:46147 (local address 127.4.207.62:35521)
0923 22:17:45.298124 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.298127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.298350 (+ 223us) server_negotiation.cc:407] Connection header received
0923 22:17:45.298843 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.298844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.298855 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.298912 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.299969 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.300474 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.301567 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.301751 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.301805 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.304391 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:45.304392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:45.304393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:45.304399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:45.305661 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:45.306022 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:45.306023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:45.306023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:45.306071 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:45.306266 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:45.306266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:45.306267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:45.306365 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:45.306413 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:45.306646 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 22:17:45.306718 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:17:45.307005 13591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.298031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:46147)
0923 22:17:45.298212 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.298260 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.298757 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.298982 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.298985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.299667 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.299879 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.299883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.300606 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.300607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.301481 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.301484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.301824 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.301824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.301928 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.302520 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.302529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.304298 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:17:45.305733 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:45.305738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:45.305740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:45.305959 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:45.306123 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:17:45.306123 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:17:45.306123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:17:45.306218 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:17:45.306414 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:17:45.306418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:17:45.306539 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 22:17:45.306623 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:17:45.306747 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":131,"threads_started":1}
I20260923 22:17:45.307521 13589 heartbeater.cc:347] Connected to a master server at 127.4.207.62:35521
I20260923 22:17:45.307626 13589 heartbeater.cc:464] Registering TS with master...
I20260923 22:17:45.307806 13589 heartbeater.cc:511] Master 127.4.207.62:35521 requested a full tablet report, sending...
I20260923 22:17:45.308180 13130 ts_manager.cc:194] Registered new tserver with Master: bfcedf640b93495186de562b998b5521 (127.4.207.3:45695)
I20260923 22:17:45.308574 13130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:46147
I20260923 22:17:45.314738 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab.
Entry for principal oryx/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab.
WARNING: no policy specified for oryx/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab.
Entry for principal oryx/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab.
WARNING: no policy specified for oryx/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab.
Entry for principal oryx/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab.
I20260923 22:17:45.390612 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 13192
I20260923 22:17:45.395720 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 13326
I20260923 22:17:45.401238 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 13460
I20260923 22:17:45.405711 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 13098
I20260923 22:17:45.410672 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35521
--webserver_interface=127.4.207.62
--webserver_port=36621
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:17:45.494455 13598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:45.494740 13598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:45.494782 13598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:45.496112 13598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:17:45.496181 13598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:45.496204 13598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:45.496227 13598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:17:45.496266 13598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:17:45.497849 13598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=36621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:45.498158 13598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:45.498574 13598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:45.503675 13604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.503679 13603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.503705 13606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:45.503917 13598 server_base.cc:1034] running on GCE node
I20260923 22:17:45.504179 13598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:45.504489 13598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:45.505630 13598 hybrid_clock.cc:648] HybridClock initialized: now 1790201865505620 us; error 29 us; skew 500 ppm
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201865, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:45.508199 13598 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:17:45.508651 13598 webserver.cc:460] Webserver started at http://127.4.207.62:36621/ using document root <none> and password file <none>
I20260923 22:17:45.508905 13598 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:45.508985 13598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:45.510339 13598 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:45.510978 13613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.511153 13598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:45.511257 13598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "af241fb6e3a74c0191f0004360b9550a"
format_stamp: "Formatted at 2026-09-23 22:17:44 on dist-test-slave-7511"
server_key: "ccbaa7b17ff5fe1f922386c398cff86c"
server_key_iv: "1cafcca1ad95becfc39730833b80d884"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.511699 13598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:45.528038 13598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:45.530118 13598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:45.530334 13598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:45.534512 13598 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:35521
I20260923 22:17:45.534560 13665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:35521 every 8 connection(s)
I20260923 22:17:45.534965 13598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:17:45.535499 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 13598
I20260923 22:17:45.535770 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:34617
--local_ip_for_outbound_sockets=127.4.207.1
--tserver_master_addrs=127.4.207.62:35521
--webserver_port=39789
--webserver_interface=127.4.207.1
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.207.1
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:17:45.536662 13666 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:17:45.537669 13666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Bootstrap starting.
I20260923 22:17:45.546134 13666 log.cc:824] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:17:45.548202 13666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:17:45.548527 13666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Bootstrap complete.
I20260923 22:17:45.550851 13666 raft_consensus.cc:348] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } }
I20260923 22:17:45.551084 13666 raft_consensus.cc:729] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af241fb6e3a74c0191f0004360b9550a, State: Initialized, Role: FOLLOWER
I20260923 22:17:45.551267 13666 consensus_queue.cc:260] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } }
I20260923 22:17:45.551344 13666 raft_consensus.cc:388] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:17:45.551398 13666 raft_consensus.cc:482] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:17:45.551465 13666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:17:45.552639 13666 raft_consensus.cc:504] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } }
I20260923 22:17:45.552745 13666 leader_election.cc:302] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af241fb6e3a74c0191f0004360b9550a; no voters:
I20260923 22:17:45.552953 13666 leader_election.cc:288] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 22:17:45.552992 13670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:17:45.553120 13670 raft_consensus.cc:686] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [term 2 LEADER]: Becoming Leader. State: Replica: af241fb6e3a74c0191f0004360b9550a, State: Running, Role: LEADER
I20260923 22:17:45.553203 13670 consensus_queue.cc:237] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } }
I20260923 22:17:45.553221 13666 sys_catalog.cc:564] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:17:45.553483 13672 sys_catalog.cc:455] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "af241fb6e3a74c0191f0004360b9550a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } } }
I20260923 22:17:45.553574 13672 sys_catalog.cc:458] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:45.553815 13674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:17:45.554376 13674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:17:45.554541 13674 catalog_manager.cc:1259] Loaded cluster ID: f77f5524424a493399bbfa8b6cddddc8
I20260923 22:17:45.554589 13674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:17:45.554850 13671 sys_catalog.cc:455] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: SysCatalogTable state changed. Reason: New leader af241fb6e3a74c0191f0004360b9550a. Latest consensus state: current_term: 2 leader_uuid: "af241fb6e3a74c0191f0004360b9550a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af241fb6e3a74c0191f0004360b9550a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35521 } } }
I20260923 22:17:45.554924 13671 sys_catalog.cc:458] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a [sys.catalog]: This master's current role is: LEADER
I20260923 22:17:45.555804 13674 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:17:45.556000 13674 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af241fb6e3a74c0191f0004360b9550a: Loaded TSK: 0
I20260923 22:17:45.556202 13674 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:17:45.625767 13668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:45.626020 13668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:45.626060 13668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:45.627233 13668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:45.627303 13668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:45.627350 13668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:17:45.628860 13668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.1.keytab
--principal=oryx/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:34617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=39789
--tserver_master_addrs=127.4.207.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:45.629118 13668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:45.629347 13668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:45.634397 13692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:45.634524 13668 server_base.cc:1034] running on GCE node
W20260923 22:17:45.634397 13693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.634466 13695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:45.634851 13668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:45.635116 13668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:45.636269 13668 hybrid_clock.cc:648] HybridClock initialized: now 1790201865636255 us; error 31 us; skew 500 ppm
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:45.638994 13668 init.cc:377] Logged in from keytab as oryx/127.4.207.1@KRBTEST.COM (short username oryx)
I20260923 22:17:45.639582 13668 webserver.cc:460] Webserver started at http://127.4.207.1:39789/ using document root <none> and password file <none>
I20260923 22:17:45.639880 13668 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:45.639976 13668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:45.641831 13668 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:17:45.642532 13702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.642724 13668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:45.642834 13668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "56148672e32341d59016b05942e17e12"
format_stamp: "Formatted at 2026-09-23 22:17:44 on dist-test-slave-7511"
server_key: "8dc59e8cdb620475d6aec5f8d5af55a3"
server_key_iv: "096e5a8bd50d7d09421f7afc0f01a0a1"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.643319 13668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:45.657096 13668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:45.659829 13668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:45.660014 13668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:45.660326 13668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:45.660748 13668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:45.660804 13668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.660840 13668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:45.660871 13668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.669044 13668 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:34617
I20260923 22:17:45.669093 13815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:34617 every 8 connection(s)
I20260923 22:17:45.669651 13668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:17:45.671128 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 13668
I20260923 22:17:45.671459 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:37265
--local_ip_for_outbound_sockets=127.4.207.2
--tserver_master_addrs=127.4.207.62:35521
--webserver_port=46579
--webserver_interface=127.4.207.2
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.207.2
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.679091 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.670471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:35283)
0923 22:17:45.670645 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.670690 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.671372 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.672407 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.672412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.673071 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.673271 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.673276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.674425 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.674426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.675336 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.675338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.675999 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.676000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.676131 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.676689 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.676698 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.678863 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
W20260923 22:17:45.679148 13819 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.670558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:35283 (local address 127.4.207.62:35521)
0923 22:17:45.671882 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.671893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.671940 (+ 47us) server_negotiation.cc:407] Connection header received
0923 22:17:45.672087 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.672095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.672154 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.672322 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.673402 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.674318 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.675459 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.675894 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.676024 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.678993 (+ 2969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:35283: BlockingRecv error: recv got EOF from 127.4.207.1:35283 (error 108)
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":97,"threads_started":1}
W20260923 22:17:45.679380 13816 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:35521 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.207.62:35521: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.683996 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.679642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:55315)
0923 22:17:45.679730 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.679740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.679791 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.679970 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.679971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.680083 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.680129 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.680131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.680777 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.680778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.681332 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.681333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.681696 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.681696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.681762 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.682203 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.682209 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.683907 (+ 1698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260923 22:17:45.684048 13819 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.679765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:55315 (local address 127.4.207.62:35521)
0923 22:17:45.679816 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.679818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.679828 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:45.679855 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.679856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.679870 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.679950 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.680206 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.680697 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.681442 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.681628 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.681699 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.683984 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:55315: BlockingRecv error: recv got EOF from 127.4.207.1:55315 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.688627 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.684507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:48531)
0923 22:17:45.684584 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.684595 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.684642 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.684756 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.684757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.684883 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.684910 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.684911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.685580 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.685581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.686103 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.686105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.686457 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.686458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.686510 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.686922 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.686928 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.688541 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260923 22:17:45.688673 13819 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.684548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:48531 (local address 127.4.207.62:35521)
0923 22:17:45.684587 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.684589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.684626 (+ 37us) server_negotiation.cc:407] Connection header received
0923 22:17:45.684697 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.684697 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.684707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.684760 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.684986 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.685509 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.686197 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.686383 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.686449 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.688609 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:48531: BlockingRecv error: recv got EOF from 127.4.207.1:48531 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 22:17:45.688997 13816 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:17:45.770609 13820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:45.770839 13820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:45.770866 13820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:45.771973 13820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:45.772017 13820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:45.772044 13820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:17:45.773427 13820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.2.keytab
--principal=oryx/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:37265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=46579
--tserver_master_addrs=127.4.207.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:45.773617 13820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:45.773816 13820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:45.778858 13825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.779004 13826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.778858 13828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:45.779088 13820 server_base.cc:1034] running on GCE node
I20260923 22:17:45.779269 13820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:45.779505 13820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:45.780675 13820 hybrid_clock.cc:648] HybridClock initialized: now 1790201865780662 us; error 29 us; skew 500 ppm
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:45.783315 13820 init.cc:377] Logged in from keytab as oryx/127.4.207.2@KRBTEST.COM (short username oryx)
I20260923 22:17:45.783771 13820 webserver.cc:460] Webserver started at http://127.4.207.2:46579/ using document root <none> and password file <none>
I20260923 22:17:45.783965 13820 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:45.784019 13820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:45.785283 13820 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:17:45.785861 13835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.786022 13820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:17:45.786115 13820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "574ad5fad65c42b48a7b84870a836219"
format_stamp: "Formatted at 2026-09-23 22:17:45 on dist-test-slave-7511"
server_key: "4b48b98f5f92b7813015cf7ad31a1513"
server_key_iv: "7abf33782b46b653eeaa240588c0680c"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.786466 13820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:45.804982 13820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:45.807260 13820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:45.807411 13820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:45.807708 13820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:45.808014 13820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:45.808063 13820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.808104 13820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:45.808130 13820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.813656 13820 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:37265
I20260923 22:17:45.813730 13948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:37265 every 8 connection(s)
I20260923 22:17:45.814085 13820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:17:45.817128 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 13820
I20260923 22:17:45.817416 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:45695
--local_ip_for_outbound_sockets=127.4.207.3
--tserver_master_addrs=127.4.207.62:35521
--webserver_port=37693
--webserver_interface=127.4.207.3
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.207.3
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
W20260923 22:17:45.821975 13819 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.814838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:37855 (local address 127.4.207.62:35521)
0923 22:17:45.814927 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.814929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.815172 (+ 243us) server_negotiation.cc:407] Connection header received
0923 22:17:45.815802 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.815803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.815822 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.815877 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.816784 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.817537 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.818552 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.818758 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.818841 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.821919 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:37855: BlockingRecv error: recv got EOF from 127.4.207.2:37855 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 22:17:45.822041 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.814845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:37855)
0923 22:17:45.815049 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.815084 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.815690 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.815934 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.815937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.816500 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.816708 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.816712 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.817658 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.817659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.818459 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.818461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.818852 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.818853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.818971 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.819542 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.819551 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.821837 (+ 2286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":109,"threads_started":1}
W20260923 22:17:45.822270 13949 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:35521 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.207.62:35521: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.826799 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.822520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:36107)
0923 22:17:45.822553 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.822563 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.822618 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.822739 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.822740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.822862 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.822903 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.822904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.823541 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.823542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.824044 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.824045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.824405 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.824405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.824468 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.824909 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.824914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.826717 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260923 22:17:45.826848 13819 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.822577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:36107 (local address 127.4.207.62:35521)
0923 22:17:45.822639 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.822641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.822649 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:17:45.822679 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.822680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.822689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.822738 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.822981 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.823482 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.824150 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.824340 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.824408 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.826784 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:36107: BlockingRecv error: recv got EOF from 127.4.207.2:36107 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.831280 13951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.827145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:60305)
0923 22:17:45.827194 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.827206 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.827250 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.827384 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.827385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.827544 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.827570 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.827572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.828190 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.828191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.828686 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.828687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.829026 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.829027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.829078 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.829483 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.829488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.831211 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 22:17:45.831353 13819 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.827224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:60305 (local address 127.4.207.62:35521)
0923 22:17:45.827258 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.827260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.827270 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:45.827302 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.827303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.827311 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.827367 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.827638 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.828093 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.828760 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.828940 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.829028 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.831285 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:60305: BlockingRecv error: recv got EOF from 127.4.207.2:60305 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:17:45.831455 13949 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:17:45.901152 13952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:17:45.901362 13952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:17:45.901388 13952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:17:45.902587 13952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:17:45.902628 13952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:17:45.902655 13952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:17:45.904100 13952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/krb5kdc/oryx_127.4.207.3.keytab
--principal=oryx/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:45695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=37693
--tserver_master_addrs=127.4.207.62:35521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:17:45.904315 13952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:17:45.904497 13952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:17:45.909643 13958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.909852 13960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:17:45.909837 13957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:17:45.909791 13952 server_base.cc:1034] running on GCE node
I20260923 22:17:45.910168 13952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:17:45.910459 13952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:17:45.911608 13952 hybrid_clock.cc:648] HybridClock initialized: now 1790201865911594 us; error 28 us; skew 500 ppm
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201865, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:17:45.914333 13952 init.cc:377] Logged in from keytab as oryx/127.4.207.3@KRBTEST.COM (short username oryx)
I20260923 22:17:45.914851 13952 webserver.cc:460] Webserver started at http://127.4.207.3:37693/ using document root <none> and password file <none>
I20260923 22:17:45.915105 13952 fs_manager.cc:359] Metadata directory not provided
I20260923 22:17:45.915194 13952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:17:45.916615 13952 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:17:45.917156 13967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.917347 13952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:17:45.917445 13952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "bfcedf640b93495186de562b998b5521"
format_stamp: "Formatted at 2026-09-23 22:17:45 on dist-test-slave-7511"
server_key: "5b27492a415be4202fc763f833a2b8fb"
server_key_iv: "22fe340af4558130897877c7baa593c1"
server_key_version: "encryptionkey@0"
I20260923 22:17:45.917804 13952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:17:45.925549 13952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:17:45.927603 13952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:17:45.927769 13952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:17:45.928000 13952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:17:45.928318 13952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:17:45.928369 13952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.928408 13952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:17:45.928437 13952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:17:45.934566 13952 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:45695
I20260923 22:17:45.934617 14080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:45695 every 8 connection(s)
I20260923 22:17:45.934980 13952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.942373 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 13952
I20260923 22:17:45.942684 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.935774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:49171)
0923 22:17:45.935951 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.936016 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.936552 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.936819 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.936823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.937392 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.937588 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.937592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.938298 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.938299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.939117 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.939119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.939558 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.939559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.939685 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.940174 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.940184 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.942468 (+ 2284us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
W20260923 22:17:45.942714 13819 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.935801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:49171 (local address 127.4.207.62:35521)
0923 22:17:45.935871 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.935885 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.936108 (+ 223us) server_negotiation.cc:407] Connection header received
0923 22:17:45.936667 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.936668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.936681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.936725 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.937688 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.938200 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.939213 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.939426 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.939539 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.942623 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:49171: BlockingRecv error: recv got EOF from 127.4.207.3:49171 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260923 22:17:45.942976 14081 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:35521 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.207.62:35521: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.948027 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.943219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:58523)
0923 22:17:45.943269 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.943279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.943329 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.943536 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.943537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.943651 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.943699 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.943701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.944498 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.944499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.945001 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.945003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.945375 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.945376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.945440 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.945909 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.945914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.947935 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 22:17:45.948087 14085 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.943280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:58523 (local address 127.4.207.62:35521)
0923 22:17:45.943388 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.943391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.943398 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:17:45.943418 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.943419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.943430 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.943530 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.943787 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.944419 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.945094 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.945281 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.945378 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.948011 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:58523: BlockingRecv error: recv got EOF from 127.4.207.3:58523 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260923 22:17:45.950950 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.942933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.4.207.62:35521)
0923 22:17:45.942977 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.942979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.943026 (+ 47us) server_negotiation.cc:407] Connection header received
0923 22:17:45.943282 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.943284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.943295 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.943347 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.944135 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.944565 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.945283 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.945463 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.945532 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.946885 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:17:45.946891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:17:45.946891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:17:45.946906 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:17:45.949501 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:45.949923 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:45.949927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:45.949927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:45.950025 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:17:45.950234 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:17:45.950235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:17:45.950235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:17:45.950502 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:17:45.950642 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 22:17:45.950737 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 22:17:45.950861 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:45 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:45.953280 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:45.948352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:46365)
0923 22:17:45.948398 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:45.948407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:45.948446 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:45.948615 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:45.948616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:45.948736 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:45.948770 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.948772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.949640 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.949641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.950393 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:45.950395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.950848 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.950849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:45.950972 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.951399 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:45.951404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:45.953200 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 22:17:45.953423 14081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:17:45.953438 14085 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:45.948413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:46365 (local address 127.4.207.62:35521)
0923 22:17:45.948458 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:45.948460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:45.948488 (+ 28us) server_negotiation.cc:407] Connection header received
0923 22:17:45.948510 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:45.948512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:45.948524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:45.948592 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:45.948841 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.949568 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.950458 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:45.950754 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:45.950822 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:45.953364 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:46365: BlockingRecv error: recv got EOF from 127.4.207.3:46365 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 22:17:46 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:46 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:46.694276 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:46.689508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:51747)
0923 22:17:46.689678 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:46.689693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:46.689769 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:46.689958 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:46.689959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:46.690118 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:46.690170 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.690171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.690825 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.690826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.691469 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.691471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.691852 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.691853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.691899 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.692385 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:46.692390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:46.694160 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
W20260923 22:17:46.694386 14087 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:46.689651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:51747 (local address 127.4.207.62:35521)
0923 22:17:46.689790 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:46.689792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:46.689801 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:46.689827 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:46.689827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:46.689872 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:46.689943 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:46.690252 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.690745 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.691557 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.691768 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.691833 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.694279 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:51747: BlockingRecv error: recv got EOF from 127.4.207.1:51747 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
Sep 23 22:17:46 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:46 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:46.836398 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:46.831954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:51995)
0923 22:17:46.832081 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:46.832092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:46.832154 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:46.832289 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:46.832290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:46.832436 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:46.832484 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.832486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.833087 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.833088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.833634 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.833636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.833953 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.833953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.833990 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.834506 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:46.834511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:46.836303 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
W20260923 22:17:46.836607 14087 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:46.832019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:51995 (local address 127.4.207.62:35521)
0923 22:17:46.832089 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:46.832092 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:46.832151 (+ 59us) server_negotiation.cc:407] Connection header received
0923 22:17:46.832218 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:46.832219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:46.832230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:46.832276 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:46.832545 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.833026 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.833714 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.833895 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.833947 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.836434 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:51995: BlockingRecv error: recv got EOF from 127.4.207.2:51995 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 23 22:17:46 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:46 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:46.958645 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:46.953874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:51741)
0923 22:17:46.954079 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:46.954093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:46.954213 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:46.954411 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:46.954412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:46.954575 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:46.954625 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.954627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.955381 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.955382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.955933 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:46.955935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.956234 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.956235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:46.956273 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.956735 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:46.956740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:46.958574 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":116,"threads_started":1}
W20260923 22:17:46.958781 14087 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:46.954001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:51741 (local address 127.4.207.62:35521)
0923 22:17:46.954084 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:46.954086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:46.954193 (+ 107us) server_negotiation.cc:407] Connection header received
0923 22:17:46.954317 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:46.954319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:46.954344 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:46.954386 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:46.954709 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.955297 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.956001 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:46.956176 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:46.956227 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:46.958690 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:51741: BlockingRecv error: recv got EOF from 127.4.207.3:51741 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 22:17:47 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:47 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:47.708920 14090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:47.694901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:47181)
0923 22:17:47.695054 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:47.695066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:47.695116 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:47.695258 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:47.695258 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:47.695425 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:47.695515 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.695516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.696303 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.696332 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.697044 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.697046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.697383 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.697384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.697437 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.706489 (+ 9052us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:47.706496 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:47.708796 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260923 22:17:47.708971 14091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:47.695000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:47181 (local address 127.4.207.62:35521)
0923 22:17:47.695123 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:47.695125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:47.695135 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:47.695162 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:47.695163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:47.695173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:47.695243 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:47.695607 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.696192 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.697112 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.697322 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.697379 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.708897 (+ 11518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:47181: BlockingRecv error: recv got EOF from 127.4.207.1:47181 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
Sep 23 22:17:47 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:47 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:47.841821 14092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:47.836974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:46215)
0923 22:17:47.837122 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:47.837134 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:47.837204 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:47.837338 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:47.837339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:47.837496 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:47.837547 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.837549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.838255 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.838256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.838981 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.838984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.839362 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.839363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.839418 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.839895 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:47.839900 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:47.841701 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
W20260923 22:17:47.841850 14091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:47.837063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:46215 (local address 127.4.207.62:35521)
0923 22:17:47.837141 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:47.837142 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:47.837201 (+ 59us) server_negotiation.cc:407] Connection header received
0923 22:17:47.837265 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:47.837266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:47.837278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:47.837329 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:47.837619 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.838171 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.839083 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.839277 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.839358 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.841789 (+ 2431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:46215: BlockingRecv error: recv got EOF from 127.4.207.2:46215 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 22:17:47 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:47 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:47.963763 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:47.959161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:52361)
0923 22:17:47.959306 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:47.959318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:47.959380 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:47.959552 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:47.959553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:47.959727 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:47.959777 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.959778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.960390 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.960391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.960915 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:47.960917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.961286 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.961288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:47.961327 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.961833 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:47.961843 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:47.963684 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
W20260923 22:17:47.963913 14091 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:47.959263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:52361 (local address 127.4.207.62:35521)
0923 22:17:47.959325 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:47.959327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:47.959366 (+ 39us) server_negotiation.cc:407] Connection header received
0923 22:17:47.959473 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:47.959474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:47.959486 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:47.959532 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:47.959838 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.960326 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.961013 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:47.961205 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:47.961269 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:47.963825 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:52361: BlockingRecv error: recv got EOF from 127.4.207.3:52361 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 23 22:17:48 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:48 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:48.714284 14094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:48.709419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:45719)
0923 22:17:48.709561 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:48.709574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:48.709639 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:48.709795 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:48.709796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:48.709974 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:48.710019 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.710021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.710724 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.710725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.711285 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.711287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.711740 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.711741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.711798 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.712245 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:48.712249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:48.714217 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260923 22:17:48.714394 14095 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:48.709465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:45719 (local address 127.4.207.62:35521)
0923 22:17:48.709565 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:48.709567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:48.709619 (+ 52us) server_negotiation.cc:407] Connection header received
0923 22:17:48.709695 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:48.709695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:48.709708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:48.709789 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:48.710110 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.710637 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.711370 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.711650 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.711738 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.714315 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:45719: BlockingRecv error: recv got EOF from 127.4.207.1:45719 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
Sep 23 22:17:48 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:48 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:48.847013 14096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:48.842323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:39983)
0923 22:17:48.842501 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:48.842512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:48.842600 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:48.842762 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:48.842763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:48.842963 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:48.843017 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.843019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.843756 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.843757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.844294 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.844296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.844644 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.844645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.844700 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.845124 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:48.845129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:48.846937 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
W20260923 22:17:48.847095 14095 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:48.842406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:39983 (local address 127.4.207.62:35521)
0923 22:17:48.842483 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:48.842485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:48.842579 (+ 94us) server_negotiation.cc:407] Connection header received
0923 22:17:48.842665 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:48.842666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:48.842692 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:48.842753 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:48.843095 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.843661 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.844373 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.844563 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.844644 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.847036 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:39983: BlockingRecv error: recv got EOF from 127.4.207.2:39983 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 22:17:48 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:48 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:48.968606 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:48.964341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:59265)
0923 22:17:48.964462 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:48.964474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:48.964520 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:48.964640 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:48.964640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:48.964805 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:48.964851 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.964853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.965472 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.965473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.965998 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:48.965999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.966336 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.966337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:48.966383 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.966822 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:48.966827 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:48.968538 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
W20260923 22:17:48.968693 14095 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:48.964417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:59265 (local address 127.4.207.62:35521)
0923 22:17:48.964496 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:48.964498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:48.964523 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:17:48.964579 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:48.964580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:48.964592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:48.964634 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:48.964908 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.965408 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.966067 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:48.966260 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:48.966324 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:48.968628 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:59265: BlockingRecv error: recv got EOF from 127.4.207.3:59265 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 23 22:17:49 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:49 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:49.719322 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:49.714721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:51799)
0923 22:17:49.714900 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:49.714911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:49.714976 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:49.715101 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:49.715102 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:49.715284 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:49.715371 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.715375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.716045 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.716047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.716576 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.716578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.716893 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.716893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.716934 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.717372 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:49.717377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:49.719242 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
W20260923 22:17:49.719544 14099 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:49.714787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:51799 (local address 127.4.207.62:35521)
0923 22:17:49.714913 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:49.714915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:49.714958 (+ 43us) server_negotiation.cc:407] Connection header received
0923 22:17:49.715019 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:49.715020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:49.715030 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:49.715087 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:49.715456 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.715951 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.716653 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.716837 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.716883 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.719465 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:51799: BlockingRecv error: recv got EOF from 127.4.207.1:51799 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":47,"threads_started":1}
Sep 23 22:17:49 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:49 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:49.852030 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:49.847638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:40093)
0923 22:17:49.847759 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:49.847769 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:49.847823 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:49.847988 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:49.847989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:49.848143 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:49.848191 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.848192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.848883 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.848884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.849395 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.849397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.849709 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.849710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.849748 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.850145 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:49.850149 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:49.851962 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
W20260923 22:17:49.852118 14099 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:49.847702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:40093 (local address 127.4.207.62:35521)
0923 22:17:49.847762 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:49.847764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:49.847818 (+ 54us) server_negotiation.cc:407] Connection header received
0923 22:17:49.847891 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:49.847892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:49.847901 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:49.847976 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:49.848266 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.848813 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.849465 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.849640 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.849692 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.852057 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:40093: BlockingRecv error: recv got EOF from 127.4.207.2:40093 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 23 22:17:49 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:49 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:49.974102 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:49.969190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:39989)
0923 22:17:49.969365 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:49.969377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:49.969440 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:49.969587 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:49.969588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:49.969761 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:49.969813 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.969814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.970716 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.970717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.971332 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:49.971335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.971692 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.971693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:49.971737 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.972178 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:49.972184 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:49.974005 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
W20260923 22:17:49.974143 14099 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:49.969312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:39989 (local address 127.4.207.62:35521)
0923 22:17:49.969379 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:49.969381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:49.969424 (+ 43us) server_negotiation.cc:407] Connection header received
0923 22:17:49.969495 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:49.969495 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:49.969507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:49.969567 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:49.969885 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.970597 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.971409 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:49.971608 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:49.971680 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:49.974080 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:39989: BlockingRecv error: recv got EOF from 127.4.207.3:39989 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 22:17:50 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:50 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:50.724364 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:50.719808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:55213)
0923 22:17:50.719937 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:50.719950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:50.720021 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:50.720180 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:50.720181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:50.720348 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:50.720397 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.720399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.721126 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.721127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.721718 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.721721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.722058 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.722059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.722097 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.722504 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:50.722509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:50.724303 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260923 22:17:50.724501 14103 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:50.719879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:55213 (local address 127.4.207.62:35521)
0923 22:17:50.720016 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:50.720018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:50.720027 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:50.720089 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:50.720090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:50.720103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:50.720168 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:50.720513 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.721058 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.721807 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.721995 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.722050 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.724391 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:55213: BlockingRecv error: recv got EOF from 127.4.207.1:55213 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
Sep 23 22:17:50 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:50 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:50.856827 14104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:50.852447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:56021)
0923 22:17:50.852617 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:50.852627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:50.852679 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:50.852799 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:50.852799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:50.852952 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:50.853002 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.853004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.853750 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.853751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.854271 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.854272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.854658 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.854659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.854700 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.855101 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:50.855106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:50.856772 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
W20260923 22:17:50.856978 14103 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:50.852565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:56021 (local address 127.4.207.62:35521)
0923 22:17:50.852611 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:50.852612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:50.852681 (+ 69us) server_negotiation.cc:407] Connection header received
0923 22:17:50.852725 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:50.852726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:50.852737 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:50.852782 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:50.853060 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.853684 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.854341 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.854601 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.854652 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.856879 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:56021: BlockingRecv error: recv got EOF from 127.4.207.2:56021 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 23 22:17:50 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:50 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:50.979349 14105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:50.974608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:46019)
0923 22:17:50.974774 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:50.974786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:50.974855 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:50.974990 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:50.974991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:50.975153 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:50.975200 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.975201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.976049 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.976051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.976736 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:50.976739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.977047 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.977047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:50.977082 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.977532 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:50.977536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:50.979290 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
W20260923 22:17:50.979558 14103 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:50.974709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:46019 (local address 127.4.207.62:35521)
0923 22:17:50.974763 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:50.974765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:50.974841 (+ 76us) server_negotiation.cc:407] Connection header received
0923 22:17:50.974915 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:50.974916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:50.974929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:50.974971 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:50.975326 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.975956 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.976808 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:50.976990 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:50.977044 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:50.979419 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:46019: BlockingRecv error: recv got EOF from 127.4.207.3:46019 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 22:17:51 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:51 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:51.729197 14106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:51.724788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:48611)
0923 22:17:51.724929 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:51.724947 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:51.724995 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:51.725132 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:51.725132 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:51.725314 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:51.725364 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.725366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.726004 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.726005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.726544 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.726546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.726870 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.726871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.726911 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.727365 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:51.727369 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:51.729141 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
W20260923 22:17:51.729380 14107 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:51.724852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:48611 (local address 127.4.207.62:35521)
0923 22:17:51.724980 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:51.724982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:51.724991 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:51.725039 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:51.725040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:51.725050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:51.725120 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:51.725437 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.725936 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.726613 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.726809 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.726858 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.729306 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:48611: BlockingRecv error: recv got EOF from 127.4.207.1:48611 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
Sep 23 22:17:51 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:51 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:51.861712 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:51.857296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:57191)
0923 22:17:51.857437 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:51.857448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:51.857508 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:51.857627 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:51.857628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:51.857806 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:51.857875 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.857877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.858497 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.858498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.859016 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.859018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.859373 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.859373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.859410 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.859875 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:51.859879 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:51.861653 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
W20260923 22:17:51.861894 14107 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:51.857323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:57191 (local address 127.4.207.62:35521)
0923 22:17:51.857388 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:51.857391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:51.857509 (+ 118us) server_negotiation.cc:407] Connection header received
0923 22:17:51.857554 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:51.857555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:51.857565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:51.857612 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:51.857960 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.858434 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.859112 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.859311 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.859367 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.861773 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:57191: BlockingRecv error: recv got EOF from 127.4.207.2:57191 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 23 22:17:51 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:51 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:51.984813 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:51.979931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:54903)
0923 22:17:51.980136 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:51.980149 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:51.980220 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:51.980393 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:51.980394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:51.980554 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:51.980602 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.980603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.981222 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.981224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.981863 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:51.981865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.982203 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.982204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:51.982251 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.982678 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:51.982682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:51.984754 (+ 2072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":117,"threads_started":1}
W20260923 22:17:51.984911 14107 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:51.979996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:54903 (local address 127.4.207.62:35521)
0923 22:17:51.980071 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:51.980074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:51.980220 (+ 146us) server_negotiation.cc:407] Connection header received
0923 22:17:51.980278 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:51.980278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:51.980291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:51.980343 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:51.980670 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.981151 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.981942 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:51.982140 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:51.982195 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:51.984841 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:54903: BlockingRecv error: recv got EOF from 127.4.207.3:54903 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 23 22:17:52 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:52 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:52.743707 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:52.729761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:58005)
0923 22:17:52.729904 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:52.729917 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:52.729980 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:52.730149 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:52.730150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:52.730315 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:52.730366 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.730367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.731316 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.731317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.731939 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.731942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.732334 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.732335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.732401 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.741318 (+ 8917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:52.741326 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:52.743645 (+ 2319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
W20260923 22:17:52.743882 14111 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:52.729868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:58005 (local address 127.4.207.62:35521)
0923 22:17:52.730005 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:52.730007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:52.730017 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:52.730044 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:52.730045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:52.730055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:52.730130 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:52.730477 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.731184 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.732040 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.732248 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.732324 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.743791 (+ 11467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:58005: BlockingRecv error: recv got EOF from 127.4.207.1:58005 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
Sep 23 22:17:52 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:52 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:52.867136 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:52.862290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:51605)
0923 22:17:52.862439 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:52.862451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:52.862504 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:52.862681 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:52.862682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:52.862849 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:52.862901 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.862903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.863981 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.863982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.864520 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.864522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.864829 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.864830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.864870 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.865303 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:52.865307 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:52.867075 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
W20260923 22:17:52.867278 14111 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:52.862443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:51605 (local address 127.4.207.62:35521)
0923 22:17:52.862517 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:52.862519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:52.862555 (+ 36us) server_negotiation.cc:407] Connection header received
0923 22:17:52.862584 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:52.862585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:52.862596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:52.862641 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:52.863041 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.863911 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.864591 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.864769 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.864818 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.867194 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:51605: BlockingRecv error: recv got EOF from 127.4.207.2:51605 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 22:17:52 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:52 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:52.989925 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:52.985294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:55347)
0923 22:17:52.985450 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:52.985460 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:52.985515 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:52.985632 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:52.985633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:52.985807 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:52.985868 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.985871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.986499 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.986500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.987027 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:52.987029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.987369 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.987369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:52.987409 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.987961 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:52.987966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:52.989864 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
W20260923 22:17:52.990085 14111 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:52.985380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:55347 (local address 127.4.207.62:35521)
0923 22:17:52.985430 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:52.985432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:52.985517 (+ 85us) server_negotiation.cc:407] Connection header received
0923 22:17:52.985571 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:52.985572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:52.985583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:52.985621 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:52.985929 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.986427 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.987105 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:52.987301 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:52.987356 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:52.990000 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:55347: BlockingRecv error: recv got EOF from 127.4.207.3:55347 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 22:17:53 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:53 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:53.750588 14114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:53.744388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:55989)
0923 22:17:53.745788 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:53.745802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:53.745883 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:53.746055 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:53.746055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:53.746209 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:53.746267 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.746268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.746968 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.746970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.747697 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.747700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.748063 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.748063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.748103 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.748650 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:53.748655 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:53.750489 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1376,"thread_start_us":70,"threads_started":1}
W20260923 22:17:53.750669 14115 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:53.745651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:55989 (local address 127.4.207.62:35521)
0923 22:17:53.745815 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:53.745818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:53.745862 (+ 44us) server_negotiation.cc:407] Connection header received
0923 22:17:53.745944 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:53.745945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:53.745959 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:53.746052 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:53.746331 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.746845 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.747782 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.747991 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.748049 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.750577 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:55989: BlockingRecv error: recv got EOF from 127.4.207.1:55989 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
Sep 23 22:17:53 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:53 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:53.873081 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:53.867762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:55031)
0923 22:17:53.867972 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:53.867989 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:53.868078 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:53.868299 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:53.868300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:53.868473 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:53.868522 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.868524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.869373 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.869374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.870100 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.870103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.870517 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.870518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.870596 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.871059 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:53.871064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:53.873012 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":117,"threads_started":1}
W20260923 22:17:53.873164 14115 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:53.867835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:55031 (local address 127.4.207.62:35521)
0923 22:17:53.867922 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:53.867925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:53.868057 (+ 132us) server_negotiation.cc:407] Connection header received
0923 22:17:53.868162 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:53.868163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:53.868178 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:53.868277 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:53.868612 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.869274 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.870197 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.870402 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.870487 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.873117 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:55031: BlockingRecv error: recv got EOF from 127.4.207.2:55031 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 23 22:17:53 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:53 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:53.995257 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:53.990456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:45101)
0923 22:17:53.990641 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:53.990653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:53.990730 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:53.990869 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:53.990870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:53.991034 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:53.991083 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.991085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.991917 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.991918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.992461 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:53.992463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.992824 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.992825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:53.992874 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.993328 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:53.993332 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:53.995195 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
W20260923 22:17:53.995416 14115 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:53.990592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:45101 (local address 127.4.207.62:35521)
0923 22:17:53.990651 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:53.990653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:53.990730 (+ 77us) server_negotiation.cc:407] Connection header received
0923 22:17:53.990793 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:53.990794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:53.990807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:53.990850 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:53.991160 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.991831 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.992540 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:53.992732 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:53.992787 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:53.995337 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:45101: BlockingRecv error: recv got EOF from 127.4.207.3:45101 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 23 22:17:54 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:54 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:54.755827 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:54.751266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:42175)
0923 22:17:54.751423 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:54.751473 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:54.751555 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:54.751680 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:54.751680 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:54.751840 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:54.751894 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.751895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.752523 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.752524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.753039 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.753041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.753447 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.753448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.753511 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.753937 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:54.753941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:54.755766 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
W20260923 22:17:54.755963 14119 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:54.751357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:42175 (local address 127.4.207.62:35521)
0923 22:17:54.751492 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:54.751494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:54.751533 (+ 39us) server_negotiation.cc:407] Connection header received
0923 22:17:54.751613 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:54.751613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:54.751624 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:54.751667 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:54.751955 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.752452 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.753109 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.753297 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.753403 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.755876 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:42175: BlockingRecv error: recv got EOF from 127.4.207.1:42175 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
Sep 23 22:17:54 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:54 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:54.878557 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:54.873611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:52683)
0923 22:17:54.873810 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:54.873822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:54.873906 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:54.874069 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:54.874071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:54.874264 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:54.874314 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.874316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.875017 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.875019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.875588 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.875590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.875961 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.875963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.876017 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.876575 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:54.876580 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:54.878458 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
W20260923 22:17:54.878619 14119 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:54.873708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:52683 (local address 127.4.207.62:35521)
0923 22:17:54.873810 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:54.873813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:54.873888 (+ 75us) server_negotiation.cc:407] Connection header received
0923 22:17:54.873977 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:54.873979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:54.873991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:54.874065 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:54.874410 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.874920 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.875682 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.875884 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.875964 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.878548 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:52683: BlockingRecv error: recv got EOF from 127.4.207.2:52683 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 23 22:17:54 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:54 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:55.000422 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:54.995760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:38031)
0923 22:17:54.995900 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:54.995919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:54.995969 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:54.996114 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:54.996115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:54.996284 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:54.996330 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.996331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.996985 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.996986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.997519 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:54.997521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.997854 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.997855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:54.997909 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:54.998425 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:54.998430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:55.000360 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
W20260923 22:17:55.000581 14119 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:54.995893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:38031 (local address 127.4.207.62:35521)
0923 22:17:54.995966 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:54.995968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:54.995977 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:17:54.996027 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:54.996027 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:54.996039 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:54.996098 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:54.996400 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.996912 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.997594 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:54.997788 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:54.997841 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.000495 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:38031: BlockingRecv error: recv got EOF from 127.4.207.3:38031 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 22:17:55 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:55 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:55.760993 14122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:55.756338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:42615)
0923 22:17:55.756497 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:55.756509 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:55.756581 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:55.756708 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:55.756708 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:55.756870 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:55.756921 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.756922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.757562 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.757562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.758079 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.758080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.758415 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.758416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.758457 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.758977 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:55.758983 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:55.760877 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
W20260923 22:17:55.761039 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:55.756384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:42615 (local address 127.4.207.62:35521)
0923 22:17:55.756505 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:55.756507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:55.756564 (+ 57us) server_negotiation.cc:407] Connection header received
0923 22:17:55.756629 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:55.756630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:55.756640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:55.756698 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:55.756985 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.757489 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.758149 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.758350 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.758403 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.760962 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:42615: BlockingRecv error: recv got EOF from 127.4.207.1:42615 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
Sep 23 22:17:55 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:55 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:55.884160 14124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:55.879123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:43161)
0923 22:17:55.879247 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:55.879261 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:55.879335 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:55.879569 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:55.879570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:55.879735 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:55.879794 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.879796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.880474 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.880475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.881006 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:55.881007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.881436 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.881437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:55.881485 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.882026 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:55.882031 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:55.884087 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
W20260923 22:17:55.884261 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:55.879216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:43161 (local address 127.4.207.62:35521)
0923 22:17:55.879312 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:55.879314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:55.879326 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:55.879397 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:55.879398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:55.879410 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:55.879503 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:55.879882 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.880404 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.881098 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:55.881356 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:55.881418 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:55.884185 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:43161: BlockingRecv error: recv got EOF from 127.4.207.2:43161 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 22:17:55 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:55 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:56.005975 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.000979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:34275)
0923 22:17:56.001147 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:56.001180 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:56.001248 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:56.001408 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:56.001409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:56.001601 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:56.001649 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.001651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.002383 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.002384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.002920 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.002922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.003308 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.003309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.003375 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.004083 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:56.004087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:56.005901 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
W20260923 22:17:56.006204 14123 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:56.001131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:34275 (local address 127.4.207.62:35521)
0923 22:17:56.001214 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.001217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.001229 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:56.001307 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.001308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.001323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.001404 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.001751 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.002268 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.003021 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.003233 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.003311 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.006144 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:34275: BlockingRecv error: recv got EOF from 127.4.207.3:34275 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 22:17:56 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:56 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:56.766590 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.761578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:52739)
0923 22:17:56.761693 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:56.761705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:56.761766 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:56.761984 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:56.761985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:56.762228 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:56.762311 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.762314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.763109 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.763110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.763761 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.763764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.764121 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.764122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.764164 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.764701 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:56.764706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:56.766517 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
W20260923 22:17:56.766692 14127 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:56.761637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:52739 (local address 127.4.207.62:35521)
0923 22:17:56.761811 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.761813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.761826 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:17:56.761858 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.761859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.761873 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.761963 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.762396 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.762963 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.763856 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.764055 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.764109 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.766616 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:52739: BlockingRecv error: recv got EOF from 127.4.207.1:52739 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
Sep 23 22:17:56 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:56 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:56.889448 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:56.884645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:37753)
0923 22:17:56.884806 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:56.884820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:56.884892 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:56.885066 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:56.885067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:56.885256 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:56.885303 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.885305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.885993 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.885995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.886540 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:56.886542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.886923 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.886924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:56.886987 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.887523 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:56.887528 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:56.889378 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
W20260923 22:17:56.889575 14127 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:56.884766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:37753 (local address 127.4.207.62:35521)
0923 22:17:56.884833 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:56.884835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:56.884871 (+ 36us) server_negotiation.cc:407] Connection header received
0923 22:17:56.884978 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:56.884979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:56.884992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:56.885061 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:56.885393 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.885899 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.886624 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:56.886839 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:56.886913 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:56.889495 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:37753: BlockingRecv error: recv got EOF from 127.4.207.2:37753 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 23 22:17:57 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:57 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:57.010933 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:57.006486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:32853)
0923 22:17:57.006635 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:57.006646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:57.006703 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:57.006839 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:57.006840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:57.007012 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:57.007059 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:57.007060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.007716 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.007716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:57.008261 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:57.008263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.008601 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.008602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:57.008642 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:57.009103 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:57.009107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:57.010857 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
W20260923 22:17:57.011047 14127 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:57.006568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:32853 (local address 127.4.207.62:35521)
0923 22:17:57.006625 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:57.006627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:57.006708 (+ 81us) server_negotiation.cc:407] Connection header received
0923 22:17:57.006750 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:57.006751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:57.006763 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:57.006816 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:57.007125 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.007646 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.008355 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.008536 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.008586 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:57.010971 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:32853: BlockingRecv error: recv got EOF from 127.4.207.3:32853 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 23 22:17:57 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:57 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:57.776782 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:57.767107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:36099)
0923 22:17:57.767251 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:57.767264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:57.767354 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:57.767529 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:57.767529 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:57.767695 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:57.767743 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:57.767745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.768448 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.768449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:57.769001 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:57.769002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.769332 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.769332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:57.769375 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:57.774885 (+ 5510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:57.774890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:57.776723 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":55,"threads_started":1}
W20260923 22:17:57.776934 14131 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:57.767198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:36099 (local address 127.4.207.62:35521)
0923 22:17:57.767338 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:57.767350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:57.767360 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:57.767416 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:57.767417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:57.767428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:57.767526 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:57.767806 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.768353 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.769078 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.769267 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.769317 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:57.776833 (+ 7516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:36099: BlockingRecv error: recv got EOF from 127.4.207.1:36099 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
Sep 23 22:17:57 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:57 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:57.894914 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:57.889906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:33565)
0923 22:17:57.890077 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:57.890092 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:57.890207 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:57.890392 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:57.890393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:57.890580 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:57.890625 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:57.890627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.891314 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.891315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:57.891873 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:57.891875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.892258 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.892259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:57.892323 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:57.892787 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:57.892792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:57.894844 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":87,"threads_started":1}
W20260923 22:17:57.895011 14131 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:57.890001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:33565 (local address 127.4.207.62:35521)
0923 22:17:57.890082 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:57.890084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:57.890185 (+ 101us) server_negotiation.cc:407] Connection header received
0923 22:17:57.890266 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:57.890267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:57.890281 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:57.890380 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:57.890712 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.891224 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.891965 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:57.892167 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:57.892255 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:57.894943 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:33565: BlockingRecv error: recv got EOF from 127.4.207.2:33565 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 22:17:58 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:58 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:58.016569 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.011558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:33677)
0923 22:17:58.011744 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.011762 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.011842 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.011985 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.011986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.012154 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:58.012199 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.012200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.012893 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.012894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.013441 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.013443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.013849 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.013850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.013901 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.014514 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:58.014519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:58.016509 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
W20260923 22:17:58.016762 14131 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:58.011694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:33677 (local address 127.4.207.62:35521)
0923 22:17:58.011775 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.011777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.011823 (+ 46us) server_negotiation.cc:407] Connection header received
0923 22:17:58.011900 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.011901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.011913 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.011965 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:58.012279 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.012796 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.013540 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.013735 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.013797 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.016678 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:33677: BlockingRecv error: recv got EOF from 127.4.207.3:33677 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 22:17:58 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:58 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:58.782166 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.777361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:58465)
0923 22:17:58.777509 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.777527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.777580 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.777735 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.777735 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.777908 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:58.777956 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.777958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.778617 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.778618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.779163 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.779164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.779650 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.779651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.779715 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.780179 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:58.780184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:58.782071 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
W20260923 22:17:58.782388 14135 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:58.777436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:58465 (local address 127.4.207.62:35521)
0923 22:17:58.777602 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.777604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.777614 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:17:58.777640 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.777640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.777650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.777722 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:58.778031 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.778544 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.779217 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.779570 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.779661 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.782303 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:58465: BlockingRecv error: recv got EOF from 127.4.207.1:58465 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
Sep 23 22:17:58 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:58 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:58.900418 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:58.895545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:59117)
0923 22:17:58.895707 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:58.895722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:58.895791 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:58.895955 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:58.895955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:58.896121 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:58.896170 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.896172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.896817 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.896818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.897357 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:58.897358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.897716 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.897717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:58.897769 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.898273 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:58.898279 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:58.900313 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
W20260923 22:17:58.900470 14135 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:58.895691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:59117 (local address 127.4.207.62:35521)
0923 22:17:58.895764 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:58.895766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:58.895778 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:17:58.895868 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:58.895869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:58.895879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:58.895932 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:58.896236 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.896746 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.897433 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:58.897621 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:58.897676 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:58.900403 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:59117: BlockingRecv error: recv got EOF from 127.4.207.2:59117 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 23 22:17:59 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:59 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:59.022415 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.017124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:56711)
0923 22:17:59.017276 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:59.017289 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:59.017352 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:59.017583 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:59.017584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:59.017809 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:59.017874 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.017877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.018761 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.018763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.019312 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.019313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.019720 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.019721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.019767 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.020319 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:59.020323 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:59.022344 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
W20260923 22:17:59.022483 14135 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:59.017207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:56711 (local address 127.4.207.62:35521)
0923 22:17:59.017394 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:59.017396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:59.017409 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:17:59.017456 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:59.017458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:59.017474 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:59.017571 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:59.017990 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.018700 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.019391 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.019654 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.019709 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.022435 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:56711: BlockingRecv error: recv got EOF from 127.4.207.3:56711 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 23 22:17:59 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:59 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:59.787400 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.782795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:47777)
0923 22:17:59.782997 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:59.783010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:59.783082 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:59.783220 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:59.783220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:59.783390 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:59.783455 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.783457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.784190 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.784191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.784719 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.784721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.785084 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.785085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.785126 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.785600 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:59.785604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:59.787329 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":109,"threads_started":1}
W20260923 22:17:59.787596 14139 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:59.782894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:47777 (local address 127.4.207.62:35521)
0923 22:17:59.783010 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:59.783011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:59.783068 (+ 57us) server_negotiation.cc:407] Connection header received
0923 22:17:59.783129 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:59.783130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:59.783142 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:59.783209 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:59.783524 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.784111 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.784807 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.785009 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.785078 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.787434 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:47777: BlockingRecv error: recv got EOF from 127.4.207.1:47777 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
Sep 23 22:17:59 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:17:59 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:17:59.905697 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:17:59.900923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:32793)
0923 22:17:59.901092 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 22:17:59.901112 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 22:17:59.901183 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:17:59.901320 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:17:59.901320 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:17:59.901491 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:17:59.901541 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.901543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.902195 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.902195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.902748 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:17:59.902750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.903149 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.903150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:17:59.903207 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.903805 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:17:59.903810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:17:59.905621 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
W20260923 22:17:59.905871 14139 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:17:59.901038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:32793 (local address 127.4.207.62:35521)
0923 22:17:59.901116 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:17:59.901117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:17:59.901166 (+ 49us) server_negotiation.cc:407] Connection header received
0923 22:17:59.901233 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:17:59.901234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:17:59.901245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:17:59.901300 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:17:59.901613 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.902122 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.902859 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:17:59.903058 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:17:59.903124 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:17:59.905767 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:32793: BlockingRecv error: recv got EOF from 127.4.207.2:32793 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 23 22:18:00 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:18:00 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.3@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:18:00.027781 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:00.022943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.3:39911)
0923 22:18:00.023123 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:00.023135 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:00.023201 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:00.023389 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:00.023390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:00.023600 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:00.023647 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:00.023648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.024324 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.024325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:00.024861 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:00.024863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.025238 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.025239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:00.025303 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:00.025809 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:00.025813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:00.027665 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
W20260923 22:18:00.027833 14139 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:18:00.023017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:39911 (local address 127.4.207.62:35521)
0923 22:18:00.023092 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:00.023095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:00.023188 (+ 93us) server_negotiation.cc:407] Connection header received
0923 22:18:00.023274 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:00.023275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:00.023296 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:00.023381 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:00.023734 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.024232 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.024962 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.025155 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.025241 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:00.027765 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.3:39911: BlockingRecv error: recv got EOF from 127.4.207.3:39911 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 22:18:00 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:18:00 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.1@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:18:00.792686 14142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:00.788047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.1:60953)
0923 22:18:00.788186 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:00.788198 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:00.788258 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:00.788401 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:00.788402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:00.788564 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:00.788616 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:00.788617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.789274 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.789274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:00.789884 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:00.789887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.790234 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.790235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:00.790289 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:00.790731 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:00.790735 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:00.792590 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
W20260923 22:18:00.792760 14143 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:18:00.788125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:60953 (local address 127.4.207.62:35521)
0923 22:18:00.788242 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:00.788246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:00.788261 (+ 15us) server_negotiation.cc:407] Connection header received
0923 22:18:00.788305 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:00.788306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:00.788316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:00.788384 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:00.788683 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.789200 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.789971 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.790171 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.790227 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:00.792688 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.1:60953: BlockingRecv error: recv got EOF from 127.4.207.1:60953 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":69,"threads_started":1}
Sep 23 22:18:00 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
Sep 23 22:18:00 dist-test-slave-7511 krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.207.2@KRBTEST.COM for oryx/127.4.207.62@KRBTEST.COM, Server not found in Kerberos database
I20260923 22:18:00.911330 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:00.906260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35521 (local address 127.4.207.2:41135)
0923 22:18:00.906460 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:00.906473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:00.906548 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:00.906678 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:00.906679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:00.906867 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:00.906933 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:00.906936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.907735 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.907736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:00.908296 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:00.908297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.908704 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.908705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:00.908764 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:00.909287 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:00.909292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:00.911268 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.207.62:35521: Server oryx/127.4.207.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":107,"threads_started":1}
W20260923 22:18:00.911527 14143 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:18:00.906365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:41135 (local address 127.4.207.62:35521)
0923 22:18:00.906457 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:00.906459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:00.906525 (+ 66us) server_negotiation.cc:407] Connection header received
0923 22:18:00.906606 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:00.906607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:00.906619 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:00.906674 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:00.907022 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.907641 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.908398 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:00.908612 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:00.908703 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:00.911413 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.207.2:41135: BlockingRecv error: recv got EOF from 127.4.207.2:41135 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 22:18:00.948192 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 13668
I20260923 22:18:00.953228 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 13820
I20260923 22:18:00.957870 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 13952
I20260923 22:18:00.962026 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 13598
2026-09-23T22:18:00Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17500 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T22:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:18:00Z Disabled control of system clock
I20260923 22:18:00.990545 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42097
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:42097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 22:18:01.074168 14148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:01.074426 14148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:01.074450 14148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:01.075802 14148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:18:01.075852 14148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:01.075865 14148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:01.075878 14148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:18:01.075891 14148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:18:01.077268 14148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:42097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:01.077512 14148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:01.077769 14148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:01.083204 14156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.083244 14148 server_base.cc:1034] running on GCE node
W20260923 22:18:01.083204 14154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.083199 14153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.083673 14148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:01.084028 14148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:01.085211 14148 hybrid_clock.cc:648] HybridClock initialized: now 1790201881085205 us; error 40 us; skew 500 ppm
I20260923 22:18:01.086643 14148 webserver.cc:460] Webserver started at http://127.4.207.62:45977/ using document root <none> and password file <none>
I20260923 22:18:01.086992 14148 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:01.087082 14148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:01.087245 14148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:01.088461 14148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "6b131c109cea41b4ba94f59801112952"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "0038ba9cfbd66da4d9b16dba467015bd"
server_key_iv: "c8a61fd494e0bf4059b6b70bf8f24f87"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.088852 14148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "6b131c109cea41b4ba94f59801112952"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "0038ba9cfbd66da4d9b16dba467015bd"
server_key_iv: "c8a61fd494e0bf4059b6b70bf8f24f87"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.090219 14148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:18:01.090976 14162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.091151 14148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:18:01.091249 14148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "6b131c109cea41b4ba94f59801112952"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "0038ba9cfbd66da4d9b16dba467015bd"
server_key_iv: "c8a61fd494e0bf4059b6b70bf8f24f87"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.091341 14148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:01.117499 14148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:01.117872 14148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:01.118062 14148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:01.122344 14148 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:42097
I20260923 22:18:01.122378 14214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:42097 every 8 connection(s)
I20260923 22:18:01.122772 14148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:18:01.123566 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:01.125805 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 14148
I20260923 22:18:01.125921 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:18:01.126124 4924 external_mini_cluster.cc:1428] Setting key 2a1290b6d1fc478ef39b47906c5a3f97
I20260923 22:18:01.126533 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952: Bootstrap starting.
I20260923 22:18:01.127255 14215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:01.127631 14215 log.cc:824] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:01.128417 14215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952: No bootstrap required, opened a new log
I20260923 22:18:01.130748 14215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b131c109cea41b4ba94f59801112952" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42097 } }
I20260923 22:18:01.130971 14215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:01.131016 14215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b131c109cea41b4ba94f59801112952, State: Initialized, Role: FOLLOWER
I20260923 22:18:01.131179 14215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b131c109cea41b4ba94f59801112952" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42097 } }
I20260923 22:18:01.131285 14215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:18:01.131319 14215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:18:01.131368 14215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:01.132117 14215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b131c109cea41b4ba94f59801112952" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42097 } }
I20260923 22:18:01.132251 14215 leader_election.cc:302] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b131c109cea41b4ba94f59801112952; no voters:
I20260923 22:18:01.132310 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.126765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.4.207.62:42097)
0923 22:18:01.126976 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:01.126984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:01.127011 (+ 27us) server_negotiation.cc:407] Connection header received
0923 22:18:01.127569 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:01.127577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:01.127648 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:01.127822 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:01.128033 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.129042 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.130210 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.130663 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.130775 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:01.131563 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:01.131567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:01.131567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:18:01.131746 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:01.131783 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:18:01.131794 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:01.131846 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:01.131953 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 22:18:01.132017 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
I20260923 22:18:01.132448 14215 leader_election.cc:288] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:18:01.132526 14220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:18:01.132741 14215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:18:01.132946 14220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [term 1 LEADER]: Becoming Leader. State: Replica: 6b131c109cea41b4ba94f59801112952, State: Running, Role: LEADER
I20260923 22:18:01.133037 14220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b131c109cea41b4ba94f59801112952" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42097 } }
I20260923 22:18:01.133364 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b131c109cea41b4ba94f59801112952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b131c109cea41b4ba94f59801112952" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42097 } } }
I20260923 22:18:01.133468 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [sys.catalog]: This master's current role is: LEADER
I20260923 22:18:01.133598 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b131c109cea41b4ba94f59801112952. Latest consensus state: current_term: 1 leader_uuid: "6b131c109cea41b4ba94f59801112952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b131c109cea41b4ba94f59801112952" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42097 } } }
I20260923 22:18:01.133662 14229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:18:01.133737 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952 [sys.catalog]: This master's current role is: LEADER
I20260923 22:18:01.134354 14229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:18:01.136091 14229 catalog_manager.cc:1347] Generated new cluster ID: dfde8e4a9e914a0cbe369dd35f97f511
I20260923 22:18:01.136161 14229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:18:01.151278 14229 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:18:01.151875 14229 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:18:01.154532 14229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b131c109cea41b4ba94f59801112952: Generated new TSK 0
I20260923 22:18:01.154732 14229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:18:01.164595 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42097
--builtin_ntp_servers=127.4.207.20:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 22:18:01.250895 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:01.251125 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:01.251150 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:01.252458 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:01.252518 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:01.252548 14239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:18:01.254009 14239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:01.254312 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:01.254658 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:01.260203 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.260185 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.260288 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.260582 14239 server_base.cc:1034] running on GCE node
I20260923 22:18:01.260838 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:01.261142 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:01.262307 14239 hybrid_clock.cc:648] HybridClock initialized: now 1790201881262348 us; error 91 us; skew 500 ppm
I20260923 22:18:01.263805 14239 webserver.cc:460] Webserver started at http://127.4.207.1:40565/ using document root <none> and password file <none>
I20260923 22:18:01.264122 14239 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:01.264197 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:01.264341 14239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:01.265472 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "380ace80e1f4413ab09090ddc52e5bbd"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "6b4f03147ced0bedec372438a70bbdbe"
server_key_iv: "261f8fc7852f487f18a98db285aff289"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.265866 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "380ace80e1f4413ab09090ddc52e5bbd"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "6b4f03147ced0bedec372438a70bbdbe"
server_key_iv: "261f8fc7852f487f18a98db285aff289"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.267234 14239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:18:01.268106 14253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.268283 14239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:18:01.268386 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "380ace80e1f4413ab09090ddc52e5bbd"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "6b4f03147ced0bedec372438a70bbdbe"
server_key_iv: "261f8fc7852f487f18a98db285aff289"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.268471 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:01.282034 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:01.282385 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:01.282541 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:01.282792 14239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:01.283164 14239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:01.283229 14239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.283272 14239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:01.283299 14239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.289548 14239 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:36073
I20260923 22:18:01.289595 14366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:36073 every 8 connection(s)
I20260923 22:18:01.290006 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:18:01.295492 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.290772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:49689 (local address 127.4.207.62:42097)
0923 22:18:01.290847 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:01.290849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:01.291114 (+ 265us) server_negotiation.cc:407] Connection header received
0923 22:18:01.291760 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:01.291762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:01.291772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:01.291832 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:01.292569 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.293080 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.294058 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.294245 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.294299 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:01.295165 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:01.295166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:01.295167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:18:01.295173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:01.295192 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:18:01.295197 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:01.295249 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:01.295349 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 22:18:01.295378 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 22:18:01.295734 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.290781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42097 (local address 127.4.207.1:49689)
0923 22:18:01.290995 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:01.291035 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:01.291660 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:01.291916 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:01.291920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:01.291928 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:01.292477 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:01.292495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.293166 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.293167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:01.293995 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:01.293997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.294324 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.294325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:01.294445 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:01.295010 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:18:01.295010 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:01.295041 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:18:01.295067 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:01.295250 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:01.295256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:01.295260 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:18:01.295353 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 22:18:01.295496 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260923 22:18:01.296208 14367 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42097
I20260923 22:18:01.296334 14367 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:01.296558 14367 heartbeater.cc:511] Master 127.4.207.62:42097 requested a full tablet report, sending...
I20260923 22:18:01.297096 14179 ts_manager.cc:194] Registered new tserver with Master: 380ace80e1f4413ab09090ddc52e5bbd (127.4.207.1:36073)
I20260923 22:18:01.297688 14179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.1:49689
I20260923 22:18:01.300199 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 14239
I20260923 22:18:01.300275 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:18:01.300423 4924 external_mini_cluster.cc:1428] Setting key 4165293e56c721c7c61d0e128d219794
I20260923 22:18:01.301313 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42097
--builtin_ntp_servers=127.4.207.20:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 22:18:01.383108 14370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:01.383376 14370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:01.383414 14370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:01.384644 14370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:01.384727 14370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:01.384773 14370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:18:01.386209 14370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:01.386519 14370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:01.386854 14370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:01.392233 14375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.392233 14376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.392462 14370 server_base.cc:1034] running on GCE node
W20260923 22:18:01.392244 14378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.392776 14370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:01.393034 14370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:01.394186 14370 hybrid_clock.cc:648] HybridClock initialized: now 1790201881394160 us; error 39 us; skew 500 ppm
I20260923 22:18:01.395483 14370 webserver.cc:460] Webserver started at http://127.4.207.2:46425/ using document root <none> and password file <none>
I20260923 22:18:01.395768 14370 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:01.395818 14370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:01.395967 14370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:01.397095 14370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "e70bb0cac08a40cc9d16250209f51f45"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "5e430612e63797863396fd18cb5516b3"
server_key_iv: "924be4e092e37fea9cd78d739b7c06fa"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.397475 14370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "e70bb0cac08a40cc9d16250209f51f45"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "5e430612e63797863396fd18cb5516b3"
server_key_iv: "924be4e092e37fea9cd78d739b7c06fa"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.398850 14370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:18:01.399668 14384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.399855 14370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:18:01.399971 14370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "e70bb0cac08a40cc9d16250209f51f45"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "5e430612e63797863396fd18cb5516b3"
server_key_iv: "924be4e092e37fea9cd78d739b7c06fa"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.400068 14370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:01.436486 14370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:01.436843 14370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:01.437017 14370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:01.437279 14370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:01.437642 14370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:01.437698 14370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.437733 14370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:01.437783 14370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.443981 14370 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:44553
I20260923 22:18:01.444038 14497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:44553 every 8 connection(s)
I20260923 22:18:01.444427 14370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:18:01.446388 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 14370
I20260923 22:18:01.446513 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:18:01.446698 4924 external_mini_cluster.cc:1428] Setting key 74692c38cc1dbdac19bcd732e17f3c99
I20260923 22:18:01.447844 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42097
--builtin_ntp_servers=127.4.207.20:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 22:18:01.451102 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.445298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:44557 (local address 127.4.207.62:42097)
0923 22:18:01.445372 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:01.445375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:01.445609 (+ 234us) server_negotiation.cc:407] Connection header received
0923 22:18:01.446103 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:01.446104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:01.446114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:01.446222 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:01.447287 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.448045 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.449584 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.449805 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.449885 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:01.450808 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:01.450810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:01.450810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:18:01.450815 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:01.450830 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:18:01.450834 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:01.450884 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:01.450989 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 22:18:01.451018 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 22:18:01.451335 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.445254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42097 (local address 127.4.207.2:44557)
0923 22:18:01.445466 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:01.445509 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:01.446015 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:01.446352 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:01.446356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:01.446365 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:01.447191 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:01.447200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.448159 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.448160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:01.449484 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:01.449487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.449906 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.449907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:01.450048 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:01.450642 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:18:01.450643 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:01.450674 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:18:01.450702 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:01.450893 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:01.450898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:01.450901 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:01.450990 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 22:18:01.451101 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":108,"threads_started":1}
I20260923 22:18:01.451961 14498 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42097
I20260923 22:18:01.452059 14498 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:01.452273 14498 heartbeater.cc:511] Master 127.4.207.62:42097 requested a full tablet report, sending...
I20260923 22:18:01.452700 14179 ts_manager.cc:194] Registered new tserver with Master: e70bb0cac08a40cc9d16250209f51f45 (127.4.207.2:44553)
I20260923 22:18:01.453272 14179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.2:44557
W20260923 22:18:01.534718 14501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:01.534952 14501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:01.534976 14501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:01.536203 14501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:01.536250 14501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:01.536283 14501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:18:01.537621 14501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:01.537851 14501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:01.538134 14501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:01.543661 14507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.543641 14509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.543685 14501 server_base.cc:1034] running on GCE node
W20260923 22:18:01.543646 14506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.544122 14501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:01.544392 14501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:01.545580 14501 hybrid_clock.cc:648] HybridClock initialized: now 1790201881545575 us; error 39 us; skew 500 ppm
I20260923 22:18:01.546919 14501 webserver.cc:460] Webserver started at http://127.4.207.3:46653/ using document root <none> and password file <none>
I20260923 22:18:01.547190 14501 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:01.547262 14501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:01.547416 14501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:01.548617 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "79c7604fd3a84c89b1adceac37ae0f7a"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "9d310432acc1e69ad6d91e9ecbb28c61"
server_key_iv: "085f16f651970d1f61c10c4e013fb545"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.548998 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "79c7604fd3a84c89b1adceac37ae0f7a"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "9d310432acc1e69ad6d91e9ecbb28c61"
server_key_iv: "085f16f651970d1f61c10c4e013fb545"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.550346 14501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:18:01.551100 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.551270 14501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:18:01.551390 14501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "79c7604fd3a84c89b1adceac37ae0f7a"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "9d310432acc1e69ad6d91e9ecbb28c61"
server_key_iv: "085f16f651970d1f61c10c4e013fb545"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.551504 14501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:01.560791 14501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:01.561133 14501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:01.561287 14501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:01.561527 14501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:01.561872 14501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:01.561926 14501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.561972 14501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:01.562000 14501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.567934 14501 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:37143
I20260923 22:18:01.567974 14628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:37143 every 8 connection(s)
I20260923 22:18:01.568367 14501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:18:01.573056 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 14501
I20260923 22:18:01.573166 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:18:01.573392 4924 external_mini_cluster.cc:1428] Setting key b71b2e1886ebccb0fcf334b4e198a64b
I20260923 22:18:01.573971 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.569152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:60013 (local address 127.4.207.62:42097)
0923 22:18:01.569217 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:01.569219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:01.569441 (+ 222us) server_negotiation.cc:407] Connection header received
0923 22:18:01.569953 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:01.569954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:01.569964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:01.570018 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:01.570742 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.571252 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.572379 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.572591 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.572649 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:01.573632 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:01.573634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:01.573635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:18:01.573641 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:01.573659 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:18:01.573664 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:01.573724 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:01.573850 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 22:18:01.573879 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:18:01.574211 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.569142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42097 (local address 127.4.207.3:60013)
0923 22:18:01.569335 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:01.569370 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:01.569858 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:01.570080 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:01.570083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:01.570092 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:01.570612 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:01.570616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.571340 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.571341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:01.572297 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:01.572300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:01.572667 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:01.572668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:01.572781 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:01.573433 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:18:01.573434 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:01.573480 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:18:01.573510 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:01.573739 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:01.573746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:01.573749 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:01.573850 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 22:18:01.573963 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260923 22:18:01.574656 14629 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42097
I20260923 22:18:01.574772 14629 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:01.574968 14629 heartbeater.cc:511] Master 127.4.207.62:42097 requested a full tablet report, sending...
I20260923 22:18:01.575336 14179 ts_manager.cc:194] Registered new tserver with Master: 79c7604fd3a84c89b1adceac37ae0f7a (127.4.207.3:37143)
I20260923 22:18:01.575742 14179 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.3:60013
I20260923 22:18:01.585461 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:18:01.587430 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 14239
W20260923 22:18:01.587471 14218 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:18:01.586876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.4.207.62:42097)
0923 22:18:01.586958 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:01.586961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:01.587064 (+ 103us) server_negotiation.cc:407] Connection header received
0923 22:18:01.587110 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:01.587111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:01.587121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:01.587169 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:01.587353 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49592: BlockingRecv error: recv got EOF from 127.0.0.1:49592 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 22:18:01.592031 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 14370
I20260923 22:18:01.596541 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 14501
I20260923 22:18:01.600914 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 14148
2026-09-23T22:18:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (637 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T22:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:18:01Z Disabled control of system clock
I20260923 22:18:01.627857 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:44397
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:44397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 22:18:01.709721 14640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:01.709947 14640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:01.709969 14640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:01.711061 14640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:18:01.711097 14640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:01.711110 14640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:01.711123 14640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:18:01.711135 14640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:18:01.712551 14640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:44397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:44397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:01.712750 14640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:01.713001 14640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:01.717880 14648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.717927 14646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.717942 14645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.718108 14640 server_base.cc:1034] running on GCE node
I20260923 22:18:01.718361 14640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:01.718686 14640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:01.719830 14640 hybrid_clock.cc:648] HybridClock initialized: now 1790201881719803 us; error 38 us; skew 500 ppm
I20260923 22:18:01.721169 14640 webserver.cc:460] Webserver started at http://127.4.207.62:45037/ using document root <none> and password file <none>
I20260923 22:18:01.721443 14640 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:01.721491 14640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:01.721661 14640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:01.722863 14640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "27ebd152d12c44ae8fb6fc1ead235704"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "e4c0890a52a1512867abc4a7f2dddbce"
server_key_iv: "7bc3c2dd3c7306ebb3927443867efde7"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.723246 14640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "27ebd152d12c44ae8fb6fc1ead235704"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "e4c0890a52a1512867abc4a7f2dddbce"
server_key_iv: "7bc3c2dd3c7306ebb3927443867efde7"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.724675 14640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:18:01.725452 14654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.725646 14640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:18:01.725742 14640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "27ebd152d12c44ae8fb6fc1ead235704"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "e4c0890a52a1512867abc4a7f2dddbce"
server_key_iv: "7bc3c2dd3c7306ebb3927443867efde7"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.725819 14640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:01.729439 14640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:01.729717 14640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:01.729862 14640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:01.734043 14640 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:44397
I20260923 22:18:01.734099 14706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:44397 every 8 connection(s)
I20260923 22:18:01.734483 14640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:18:01.735174 14707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:01.738095 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704: Bootstrap starting.
I20260923 22:18:01.738708 14707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:01.739024 14707 log.cc:824] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:01.739814 14707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704: No bootstrap required, opened a new log
I20260923 22:18:01.741433 14707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ebd152d12c44ae8fb6fc1ead235704" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44397 } }
I20260923 22:18:01.741583 14707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:01.741606 14707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27ebd152d12c44ae8fb6fc1ead235704, State: Initialized, Role: FOLLOWER
I20260923 22:18:01.741807 14707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ebd152d12c44ae8fb6fc1ead235704" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44397 } }
I20260923 22:18:01.741874 14707 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:18:01.741897 14707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:18:01.741971 14707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:01.742350 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 14640
I20260923 22:18:01.742424 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:18:01.742621 4924 external_mini_cluster.cc:1428] Setting key ceeaa320788b7b024d81ee8dd8f7f1e4
I20260923 22:18:01.742609 14707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ebd152d12c44ae8fb6fc1ead235704" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44397 } }
I20260923 22:18:01.742722 14707 leader_election.cc:302] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27ebd152d12c44ae8fb6fc1ead235704; no voters:
I20260923 22:18:01.742947 14707 leader_election.cc:288] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:18:01.742978 14710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:18:01.743222 14710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [term 1 LEADER]: Becoming Leader. State: Replica: 27ebd152d12c44ae8fb6fc1ead235704, State: Running, Role: LEADER
I20260923 22:18:01.743235 14707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:18:01.743600 14710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ebd152d12c44ae8fb6fc1ead235704" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44397 } }
I20260923 22:18:01.744289 14713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27ebd152d12c44ae8fb6fc1ead235704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ebd152d12c44ae8fb6fc1ead235704" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44397 } } }
I20260923 22:18:01.744371 14713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [sys.catalog]: This master's current role is: LEADER
I20260923 22:18:01.745206 14711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27ebd152d12c44ae8fb6fc1ead235704. Latest consensus state: current_term: 1 leader_uuid: "27ebd152d12c44ae8fb6fc1ead235704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27ebd152d12c44ae8fb6fc1ead235704" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44397 } } }
I20260923 22:18:01.745288 14711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704 [sys.catalog]: This master's current role is: LEADER
W20260923 22:18:01.745296 14728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:18:01.745381 14728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:18:01.745573 14724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:18:01.745759 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.743138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.4.207.62:44397)
0923 22:18:01.743350 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:01.743385 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:01.743575 (+ 190us) server_negotiation.cc:407] Connection header received
0923 22:18:01.743745 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:01.743751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:01.743789 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:01.743944 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:01.745129 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:01.745135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:01.745135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:18:01.745141 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:01.745206 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:18:01.745222 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:01.745315 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:01.745468 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 22:18:01.745540 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260923 22:18:01.745761 14724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:18:01.747313 14724 catalog_manager.cc:1347] Generated new cluster ID: 22f78748dc3a473cbb370d9420f9b65a
I20260923 22:18:01.747365 14724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:18:01.757458 14724 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:18:01.757661 14724 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:18:01.771742 14724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27ebd152d12c44ae8fb6fc1ead235704: Generated new TSK 0
I20260923 22:18:01.771992 14724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:18:01.779089 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:44397
--builtin_ntp_servers=127.4.207.20:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 22:18:01.862599 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:01.862851 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:01.862890 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:01.864171 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:01.864238 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:01.864281 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:18:01.865650 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:01.865921 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:01.866218 14731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:01.871170 14736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.871201 14737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.871415 14731 server_base.cc:1034] running on GCE node
W20260923 22:18:01.871201 14739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.871776 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:01.872038 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:01.873212 14731 hybrid_clock.cc:648] HybridClock initialized: now 1790201881873196 us; error 29 us; skew 500 ppm
I20260923 22:18:01.874627 14731 webserver.cc:460] Webserver started at http://127.4.207.1:35477/ using document root <none> and password file <none>
I20260923 22:18:01.874892 14731 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:01.874963 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:01.875110 14731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:01.876240 14731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "560e0483b5314d8490c282387da6e831"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "83ce0d8a53dea8a90c3331ff690ad32f"
server_key_iv: "0f52ff8ac78421627dce7f8d9b619469"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.876607 14731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "560e0483b5314d8490c282387da6e831"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "83ce0d8a53dea8a90c3331ff690ad32f"
server_key_iv: "0f52ff8ac78421627dce7f8d9b619469"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.877946 14731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:18:01.878769 14745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.878988 14731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:18:01.879088 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "560e0483b5314d8490c282387da6e831"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "83ce0d8a53dea8a90c3331ff690ad32f"
server_key_iv: "0f52ff8ac78421627dce7f8d9b619469"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.879172 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:01.882556 14731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:01.882822 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:01.882968 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:01.883168 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:01.883517 14731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:01.883579 14731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.883618 14731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:01.883646 14731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.889428 14731 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:39341
I20260923 22:18:01.889470 14858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:39341 every 8 connection(s)
I20260923 22:18:01.889782 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:18:01.892702 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.890494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:34005 (local address 127.4.207.62:44397)
0923 22:18:01.890553 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:01.890555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:01.890780 (+ 225us) server_negotiation.cc:407] Connection header received
0923 22:18:01.891246 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:01.891247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:01.891254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:01.891307 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:01.892494 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:01.892495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:01.892495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:18:01.892499 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:01.892514 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:18:01.892518 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:01.892568 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:01.892634 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 22:18:01.892647 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:18:01.892889 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:01.890477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:44397 (local address 127.4.207.1:34005)
0923 22:18:01.890657 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:01.890689 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:01.891151 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:01.891365 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:01.891369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:01.891377 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:01.892336 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:18:01.892336 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:01.892365 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:18:01.892396 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:01.892565 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:01.892567 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:01.892570 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:01.892623 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:18:01.892670 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260923 22:18:01.893401 14859 heartbeater.cc:347] Connected to a master server at 127.4.207.62:44397
I20260923 22:18:01.893502 14859 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:01.893678 14859 heartbeater.cc:511] Master 127.4.207.62:44397 requested a full tablet report, sending...
I20260923 22:18:01.894012 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 14731
I20260923 22:18:01.894125 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:18:01.894335 4924 external_mini_cluster.cc:1428] Setting key a9e427a079f4828326191bd54320f905
I20260923 22:18:01.894335 14671 ts_manager.cc:194] Registered new tserver with Master: 560e0483b5314d8490c282387da6e831 (127.4.207.1:39341)
I20260923 22:18:01.895324 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:44397
--builtin_ntp_servers=127.4.207.20:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 22:18:01.978999 14862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:01.979236 14862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:01.979261 14862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:01.980685 14862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:01.980736 14862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:01.980767 14862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:18:01.982244 14862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:01.982522 14862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:01.982834 14862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:01.988086 14870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.988225 14862 server_base.cc:1034] running on GCE node
W20260923 22:18:01.988098 14868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:01.988145 14867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:01.988566 14862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:01.988874 14862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:01.990051 14862 hybrid_clock.cc:648] HybridClock initialized: now 1790201881990039 us; error 29 us; skew 500 ppm
I20260923 22:18:01.991485 14862 webserver.cc:460] Webserver started at http://127.4.207.2:40399/ using document root <none> and password file <none>
I20260923 22:18:01.991770 14862 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:01.991842 14862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:01.992017 14862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:01.993124 14862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "8d098f9c13e74c039b0230c3d0194138"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "f7d5ca428205cebf5885ae9ede879675"
server_key_iv: "a5091d97de76f0ea1e1c3c45ab595b96"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.993508 14862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "8d098f9c13e74c039b0230c3d0194138"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "f7d5ca428205cebf5885ae9ede879675"
server_key_iv: "a5091d97de76f0ea1e1c3c45ab595b96"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.994901 14862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:18:01.995792 14876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:01.995975 14862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:18:01.996078 14862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "8d098f9c13e74c039b0230c3d0194138"
format_stamp: "Formatted at 2026-09-23 22:18:01 on dist-test-slave-7511"
server_key: "f7d5ca428205cebf5885ae9ede879675"
server_key_iv: "a5091d97de76f0ea1e1c3c45ab595b96"
server_key_version: "encryptionkey@0"
I20260923 22:18:01.996163 14862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:01.999624 14862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:01.999924 14862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:02.000069 14862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:02.000276 14862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:02.000593 14862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:02.000648 14862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:02.000689 14862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:02.000716 14862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:02.006695 14862 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:37993
I20260923 22:18:02.006736 14989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:37993 every 8 connection(s)
I20260923 22:18:02.007097 14862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:18:02.010254 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 14862
I20260923 22:18:02.010370 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:02.007887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:33883 (local address 127.4.207.62:44397)
0923 22:18:02.007950 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:02.007952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:02.008182 (+ 230us) server_negotiation.cc:407] Connection header received
0923 22:18:02.008720 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:02.008722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:02.008729 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:02.008779 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:02.010075 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:02.010077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:02.010077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:18:02.010083 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:02.010101 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:18:02.010106 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:02.010167 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:02.010272 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 22:18:02.010289 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:18:02.010411 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:18:02.010586 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:02.007856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:44397 (local address 127.4.207.2:33883)
0923 22:18:02.008053 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:02.008090 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:02.008597 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:02.008870 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:02.008873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:02.008882 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:02.009900 (+ 1018us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:18:02.009901 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:02.009932 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:18:02.009965 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:02.010178 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:02.010184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:02.010187 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:02.010284 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 22:18:02.010349 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260923 22:18:02.010627 4924 external_mini_cluster.cc:1428] Setting key ddffe068a82fe49572af84b4f4adbc5f
I20260923 22:18:02.011269 14990 heartbeater.cc:347] Connected to a master server at 127.4.207.62:44397
I20260923 22:18:02.011402 14990 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:02.011602 14990 heartbeater.cc:511] Master 127.4.207.62:44397 requested a full tablet report, sending...
I20260923 22:18:02.011793 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:44397
--builtin_ntp_servers=127.4.207.20:40237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 22:18:02.012040 14671 ts_manager.cc:194] Registered new tserver with Master: 8d098f9c13e74c039b0230c3d0194138 (127.4.207.2:37993)
W20260923 22:18:02.097882 14993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:02.098125 14993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:02.098150 14993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:02.099556 14993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:02.099625 14993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:02.099671 14993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:18:02.101164 14993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:44397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:02.101470 14993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:02.101797 14993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:02.107044 15001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:02.107040 14999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:02.107046 14998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:02.107141 14993 server_base.cc:1034] running on GCE node
I20260923 22:18:02.107553 14993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:02.107823 14993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:02.108990 14993 hybrid_clock.cc:648] HybridClock initialized: now 1790201882108978 us; error 45 us; skew 500 ppm
I20260923 22:18:02.110430 14993 webserver.cc:460] Webserver started at http://127.4.207.3:33289/ using document root <none> and password file <none>
I20260923 22:18:02.110689 14993 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:02.110761 14993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:02.110924 14993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:02.112118 14993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "8e4e5e24d5164bb6bbf8e5be9e496688"
format_stamp: "Formatted at 2026-09-23 22:18:02 on dist-test-slave-7511"
server_key: "065ff5ff32e78e37d4a27ef22baa3706"
server_key_iv: "d05325240904050cf5fd07bf464a5cf1"
server_key_version: "encryptionkey@0"
I20260923 22:18:02.112498 14993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "8e4e5e24d5164bb6bbf8e5be9e496688"
format_stamp: "Formatted at 2026-09-23 22:18:02 on dist-test-slave-7511"
server_key: "065ff5ff32e78e37d4a27ef22baa3706"
server_key_iv: "d05325240904050cf5fd07bf464a5cf1"
server_key_version: "encryptionkey@0"
I20260923 22:18:02.113917 14993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:18:02.114758 15007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:02.114943 14993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:18:02.115072 14993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "8e4e5e24d5164bb6bbf8e5be9e496688"
format_stamp: "Formatted at 2026-09-23 22:18:02 on dist-test-slave-7511"
server_key: "065ff5ff32e78e37d4a27ef22baa3706"
server_key_iv: "d05325240904050cf5fd07bf464a5cf1"
server_key_version: "encryptionkey@0"
I20260923 22:18:02.115159 14993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:02.118711 14993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:02.119009 14993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:02.119159 14993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:02.119379 14993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:02.119756 14993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:02.119813 14993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:02.119845 14993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:02.119876 14993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:02.125737 14993 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:38067
I20260923 22:18:02.125800 15120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:38067 every 8 connection(s)
I20260923 22:18:02.126230 14993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:18:02.126665 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 14993
I20260923 22:18:02.126757 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:18:02.126941 4924 external_mini_cluster.cc:1428] Setting key 2c75dfd518cda41dfe8854d801801d2c
I20260923 22:18:02.129748 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:02.127043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:36627 (local address 127.4.207.62:44397)
0923 22:18:02.127117 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:02.127120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:02.127419 (+ 299us) server_negotiation.cc:407] Connection header received
0923 22:18:02.128118 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:02.128119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:02.128125 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:02.128190 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:02.129524 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:02.129525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:02.129525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:18:02.129530 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:02.129545 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:18:02.129549 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:02.129603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:02.129681 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 22:18:02.129694 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:18:02.130009 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:02.127062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:44397 (local address 127.4.207.3:36627)
0923 22:18:02.127259 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:02.127308 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:02.127988 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:02.128268 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:02.128272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:02.128282 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:02.129334 (+ 1052us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:18:02.129335 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:02.129367 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:18:02.129397 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:02.129617 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:02.129621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:02.129624 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:02.129691 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 22:18:02.129745 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260923 22:18:02.130534 15121 heartbeater.cc:347] Connected to a master server at 127.4.207.62:44397
I20260923 22:18:02.130614 15121 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:02.130784 15121 heartbeater.cc:511] Master 127.4.207.62:44397 requested a full tablet report, sending...
I20260923 22:18:02.131157 14671 ts_manager.cc:194] Registered new tserver with Master: 8e4e5e24d5164bb6bbf8e5be9e496688 (127.4.207.3:38067)
I20260923 22:18:02.138952 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 22:18:02.141026 14714 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:18:02.140519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42930 (local address 127.4.207.62:44397)
0923 22:18:02.140596 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:02.140599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:02.140674 (+ 75us) server_negotiation.cc:407] Connection header received
0923 22:18:02.140725 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:02.140725 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:02.140731 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:02.140781 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:02.140934 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42930: BlockingRecv error: recv got EOF from 127.0.0.1:42930 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:18:02.141039 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 14731
I20260923 22:18:02.146265 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 14862
I20260923 22:18:02.151180 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 14993
I20260923 22:18:02.155489 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 14640
2026-09-23T22:18:02Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (554 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:18:02 dist-test-slave-7511 krb5kdc[15131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:18:02 dist-test-slave-7511 krb5kdc[15131](info): set up 2 sockets
Sep 23 22:18:02 dist-test-slave-7511 krb5kdc[15131](info): commencing operation
krb5kdc: starting...
W20260923 22:18:03.188488 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:18:03.311129 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39799
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:39799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:03.393359 15144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:03.393606 15144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:03.393648 15144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:03.394786 15144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:18:03.394853 15144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:03.394877 15144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:03.394899 15144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:18:03.394927 15144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:18:03.396415 15144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:39799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:03.396698 15144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:03.396934 15144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:03.401976 15149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:03.401988 15150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:03.401976 15152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:03.402355 15144 server_base.cc:1034] running on GCE node
I20260923 22:18:03.402616 15144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:03.402953 15144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:03.404119 15144 hybrid_clock.cc:648] HybridClock initialized: now 1790201883404103 us; error 61 us; skew 500 ppm
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:03.410723 15144 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:18:03.411209 15144 webserver.cc:460] Webserver started at http://127.4.207.62:39767/ using document root <none> and password file <none>
I20260923 22:18:03.411544 15144 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:03.411619 15144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:03.411780 15144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:03.412890 15144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "4785cbd9bcc14024b9d8d7a4a8e889df"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "ae2f094b823df336a791c0924907dcdd"
server_key_iv: "689cad4649c1632c6db5e62c65b10f1d"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.413280 15144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "4785cbd9bcc14024b9d8d7a4a8e889df"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "ae2f094b823df336a791c0924907dcdd"
server_key_iv: "689cad4649c1632c6db5e62c65b10f1d"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.414659 15144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:18:03.415431 15159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.415671 15144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:18:03.415777 15144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "4785cbd9bcc14024b9d8d7a4a8e889df"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "ae2f094b823df336a791c0924907dcdd"
server_key_iv: "689cad4649c1632c6db5e62c65b10f1d"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.415874 15144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:03.426851 15144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:03.428846 15144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:03.429014 15144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:03.433043 15144 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:39799
I20260923 22:18:03.433080 15211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:39799 every 8 connection(s)
I20260923 22:18:03.433462 15144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:18:03.434428 15212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:03.436262 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 15144
I20260923 22:18:03.436379 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:18:03.436579 4924 external_mini_cluster.cc:1428] Setting key 84052361a817d91c8dbbeab8632df6f7
I20260923 22:18:03.437260 15212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df: Bootstrap starting.
I20260923 22:18:03.437929 15212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:03.438278 15212 log.cc:824] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:03.439007 15212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df: No bootstrap required, opened a new log
I20260923 22:18:03.441241 15212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4785cbd9bcc14024b9d8d7a4a8e889df" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39799 } }
I20260923 22:18:03.441434 15212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:03.441483 15212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4785cbd9bcc14024b9d8d7a4a8e889df, State: Initialized, Role: FOLLOWER
I20260923 22:18:03.441627 15212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4785cbd9bcc14024b9d8d7a4a8e889df" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39799 } }
I20260923 22:18:03.441751 15212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:18:03.441797 15212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:18:03.441876 15212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:03.442443 15212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4785cbd9bcc14024b9d8d7a4a8e889df" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39799 } }
I20260923 22:18:03.442560 15212 leader_election.cc:302] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4785cbd9bcc14024b9d8d7a4a8e889df; no voters:
I20260923 22:18:03.442751 15212 leader_election.cc:288] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:18:03.442775 15217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:18:03.442890 15217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [term 1 LEADER]: Becoming Leader. State: Replica: 4785cbd9bcc14024b9d8d7a4a8e889df, State: Running, Role: LEADER
I20260923 22:18:03.442970 15217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4785cbd9bcc14024b9d8d7a4a8e889df" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39799 } }
I20260923 22:18:03.443018 15212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:18:03.443274 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4785cbd9bcc14024b9d8d7a4a8e889df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4785cbd9bcc14024b9d8d7a4a8e889df" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39799 } } }
I20260923 22:18:03.443331 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [sys.catalog]: This master's current role is: LEADER
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:18:03.443478 15217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4785cbd9bcc14024b9d8d7a4a8e889df. Latest consensus state: current_term: 1 leader_uuid: "4785cbd9bcc14024b9d8d7a4a8e889df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4785cbd9bcc14024b9d8d7a4a8e889df" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39799 } } }
I20260923 22:18:03.443540 15217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df [sys.catalog]: This master's current role is: LEADER
I20260923 22:18:03.443939 15221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:18:03.444806 15221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:18:03.446609 15221 catalog_manager.cc:1347] Generated new cluster ID: e42718f8569a43549eda27ded9809457
I20260923 22:18:03.446664 15221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:18:03.448350 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.437106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.4.207.62:39799)
0923 22:18:03.437282 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:03.437290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:03.437315 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:18:03.437798 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:03.437805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:03.437860 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:03.437981 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:03.438788 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.439777 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.440888 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.441279 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.441365 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.444193 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:03.444197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:03.444197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:03.444209 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:03.446205 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.446625 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.446628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.446629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.446741 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.446971 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.446972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.446972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.447269 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:03.447416 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:03.447970 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 22:18:03.448128 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260923 22:18:03.461826 15221 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:18:03.462637 15221 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:18:03.471786 15221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df: Generated new TSK 0
I20260923 22:18:03.472023 15221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:18:03.503983 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39799
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:03.587862 15238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:03.588155 15238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:03.588196 15238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:03.589443 15238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:03.589520 15238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:03.589569 15238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:18:03.591109 15238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:03.591393 15238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:03.591670 15238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:03.596963 15244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:03.596972 15243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:03.597141 15238 server_base.cc:1034] running on GCE node
W20260923 22:18:03.597031 15246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:03.597416 15238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:03.597675 15238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:03.598868 15238 hybrid_clock.cc:648] HybridClock initialized: now 1790201883598854 us; error 34 us; skew 500 ppm
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:03.601432 15238 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:18:03.601925 15238 webserver.cc:460] Webserver started at http://127.4.207.1:40961/ using document root <none> and password file <none>
I20260923 22:18:03.602154 15238 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:03.602221 15238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:03.602368 15238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:03.603495 15238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "130aa307052b410ba7b200a08f25b451"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "164c7459749e8433cf7f76dc477a2763"
server_key_iv: "bc386da4efd2c5bb987555b0bf7d07c0"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.603884 15238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "130aa307052b410ba7b200a08f25b451"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "164c7459749e8433cf7f76dc477a2763"
server_key_iv: "bc386da4efd2c5bb987555b0bf7d07c0"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.605261 15238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:18:03.605997 15253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.606180 15238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:18:03.606282 15238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "130aa307052b410ba7b200a08f25b451"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "164c7459749e8433cf7f76dc477a2763"
server_key_iv: "bc386da4efd2c5bb987555b0bf7d07c0"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.606356 15238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:03.624034 15238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:03.626214 15238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:03.626336 15238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:03.626633 15238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:03.626966 15238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:03.627002 15238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.627024 15238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:03.627070 15238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.633028 15238 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:38521
I20260923 22:18:03.633104 15366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:38521 every 8 connection(s)
I20260923 22:18:03.633448 15238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:18:03.639730 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 15238
I20260923 22:18:03.639843 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:18:03.640030 4924 external_mini_cluster.cc:1428] Setting key 3c665e735eb4ae19e5555cf66d500d49
I20260923 22:18:03.643493 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.634199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:56877 (local address 127.4.207.62:39799)
0923 22:18:03.634258 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:03.634260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:03.634456 (+ 196us) server_negotiation.cc:407] Connection header received
0923 22:18:03.634944 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:03.634945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:03.634955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:03.635009 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:03.636042 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.636561 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.637534 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.637722 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.637769 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.640058 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:03.640060 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:03.640060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:03.640069 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:03.641775 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.642322 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.642324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.642324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.642395 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.642704 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.642705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.642706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.642841 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:03.642927 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:03.643250 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 22:18:03.643364 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:18:03.643695 15369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.634175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39799 (local address 127.4.207.1:56877)
0923 22:18:03.634332 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:03.634368 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:03.634858 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:03.635067 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:03.635070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:03.635747 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:03.635954 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.635960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.636644 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.636644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.637450 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.637452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.637789 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.637790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.637889 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.638462 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:03.638471 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:03.639961 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:03.641882 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:03.641890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:03.641892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:03.642206 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:03.642480 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:03.642481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:03.642482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:03.642625 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:03.642916 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:03.642923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:03.643092 (+ 169us) client_negotiation.cc:769] Sending connection context
0923 22:18:03.643214 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 22:18:03.643390 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260923 22:18:03.644296 15367 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39799
I20260923 22:18:03.644408 15367 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:03.644618 15367 heartbeater.cc:511] Master 127.4.207.62:39799 requested a full tablet report, sending...
I20260923 22:18:03.645186 15176 ts_manager.cc:194] Registered new tserver with Master: 130aa307052b410ba7b200a08f25b451 (127.4.207.1:38521)
I20260923 22:18:03.645840 15176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:56877
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:18:03.666317 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39799
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:03.748425 15372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:03.748690 15372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:03.748728 15372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:03.749928 15372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:03.750010 15372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:03.750056 15372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:18:03.751610 15372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:03.751904 15372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:03.752152 15372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:03.757382 15380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:03.757403 15377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:03.757364 15378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:03.757464 15372 server_base.cc:1034] running on GCE node
I20260923 22:18:03.757794 15372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:03.758083 15372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:03.759265 15372 hybrid_clock.cc:648] HybridClock initialized: now 1790201883759248 us; error 27 us; skew 500 ppm
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:03.761950 15372 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:18:03.762463 15372 webserver.cc:460] Webserver started at http://127.4.207.2:42185/ using document root <none> and password file <none>
I20260923 22:18:03.762719 15372 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:03.762791 15372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:03.762948 15372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:03.764113 15372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "34c703d99feb4872aee2b3fea586ee78"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "fa014de0b19871cc2f3d583fb5a5a298"
server_key_iv: "2e2f3c815da5d3a1fd9bbffd47da937e"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.764498 15372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "34c703d99feb4872aee2b3fea586ee78"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "fa014de0b19871cc2f3d583fb5a5a298"
server_key_iv: "2e2f3c815da5d3a1fd9bbffd47da937e"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.765857 15372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:18:03.766633 15387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.766808 15372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:18:03.766920 15372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "34c703d99feb4872aee2b3fea586ee78"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "fa014de0b19871cc2f3d583fb5a5a298"
server_key_iv: "2e2f3c815da5d3a1fd9bbffd47da937e"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.766997 15372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:03.774801 15372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:03.777171 15372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:03.777336 15372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:03.777561 15372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:03.777865 15372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:03.777918 15372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.777961 15372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:03.777988 15372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.784868 15372 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:43075
I20260923 22:18:03.784909 15500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:43075 every 8 connection(s)
I20260923 22:18:03.785347 15372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:18:03.791153 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 15372
I20260923 22:18:03.791265 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:18:03.791585 4924 external_mini_cluster.cc:1428] Setting key d02b67ca9bb25be6051772159f8f88b2
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:18:03.795130 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.786173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:50597 (local address 127.4.207.62:39799)
0923 22:18:03.786255 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:03.786258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:03.786414 (+ 156us) server_negotiation.cc:407] Connection header received
0923 22:18:03.786892 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:03.786893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:03.786903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:03.786958 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:03.787980 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.788499 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.789517 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.789713 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.789799 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.792184 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:03.792185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:03.792186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:03.792194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:03.793671 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.794115 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.794116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.794117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.794181 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.794448 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.794449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.794449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.794578 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:03.794648 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:03.794933 (+ 285us) server_negotiation.cc:299] Negotiation successful
0923 22:18:03.795037 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 22:18:03.795349 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.786126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39799 (local address 127.4.207.2:50597)
0923 22:18:03.786287 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:03.786324 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:03.786806 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:03.787024 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:03.787027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:03.787677 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:03.787881 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.787886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.788583 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.788583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.789411 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.789413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.789801 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.789802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.789904 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.790403 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:03.790411 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:03.792091 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:03.793781 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:03.793790 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:03.793792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:03.794035 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:03.794250 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:03.794251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:03.794251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:03.794372 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:03.794650 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:03.794657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:03.794802 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 22:18:03.794911 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 22:18:03.795087 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260923 22:18:03.795868 15501 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39799
I20260923 22:18:03.795994 15501 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:03.796181 15501 heartbeater.cc:511] Master 127.4.207.62:39799 requested a full tablet report, sending...
I20260923 22:18:03.796572 15176 ts_manager.cc:194] Registered new tserver with Master: 34c703d99feb4872aee2b3fea586ee78 (127.4.207.2:43075)
I20260923 22:18:03.796959 15176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:50597
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:18:03.819684 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39799
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:38373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:03.902542 15506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:03.902783 15506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:03.902818 15506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:03.904121 15506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:03.904175 15506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:03.904207 15506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:18:03.905819 15506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:38373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:03.906106 15506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:03.906363 15506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:03.911633 15512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:03.911657 15511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:03.911675 15506 server_base.cc:1034] running on GCE node
W20260923 22:18:03.911669 15514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:03.912083 15506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:03.912359 15506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:03.913535 15506 hybrid_clock.cc:648] HybridClock initialized: now 1790201883913520 us; error 30 us; skew 500 ppm
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:03.916175 15506 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:18:03.916657 15506 webserver.cc:460] Webserver started at http://127.4.207.3:44367/ using document root <none> and password file <none>
I20260923 22:18:03.916873 15506 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:03.916916 15506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:03.917016 15506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:18:03.918051 15506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "ba1be0fecd72461e879559048fd71575"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "47a4b1a8f3f2396b7a7c551766ead0c1"
server_key_iv: "93143a5231432934ae41d1f8ceb5ff07"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.918402 15506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "ba1be0fecd72461e879559048fd71575"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "47a4b1a8f3f2396b7a7c551766ead0c1"
server_key_iv: "93143a5231432934ae41d1f8ceb5ff07"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.919924 15506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:18:03.920640 15521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.920802 15506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:18:03.920902 15506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "ba1be0fecd72461e879559048fd71575"
format_stamp: "Formatted at 2026-09-23 22:18:03 on dist-test-slave-7511"
server_key: "47a4b1a8f3f2396b7a7c551766ead0c1"
server_key_iv: "93143a5231432934ae41d1f8ceb5ff07"
server_key_version: "encryptionkey@0"
I20260923 22:18:03.920974 15506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:03.931803 15506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:03.933740 15506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:03.933918 15506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:03.934161 15506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:03.934480 15506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:03.934514 15506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.934536 15506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:03.934548 15506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:03.940541 15506 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:37099
I20260923 22:18:03.940598 15634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:37099 every 8 connection(s)
I20260923 22:18:03.940961 15506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:18:03.944533 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 15506
I20260923 22:18:03.944671 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:18:03.944912 4924 external_mini_cluster.cc:1428] Setting key 6d8e9b82d9d8134150567f3d4cc0faeb
Sep 23 22:18:03 dist-test-slave-7511 krb5kdc[15131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201883, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:18:03.951053 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.941729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:38981 (local address 127.4.207.62:39799)
0923 22:18:03.941798 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:03.941800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:03.942040 (+ 240us) server_negotiation.cc:407] Connection header received
0923 22:18:03.942542 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:03.942544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:03.942556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:03.942668 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:03.943822 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.944504 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.945852 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.946152 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.946209 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.948499 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:03.948500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:03.948501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:03.948508 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:03.949817 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.950208 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.950209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.950210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.950261 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.950528 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.950528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.950529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.950633 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:03.950684 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:03.950923 (+ 239us) server_negotiation.cc:299] Negotiation successful
0923 22:18:03.950986 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:18:03.951293 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.941697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39799 (local address 127.4.207.3:38981)
0923 22:18:03.941883 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:03.941925 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:03.942425 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:03.942745 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:03.942749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:03.943350 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:03.943670 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.943679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.944643 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.944645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.945721 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.945724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.946226 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.946226 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.946351 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.946852 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:03.946861 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:03.948407 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:03.949895 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:03.949900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:03.949902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:03.950141 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:03.950353 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:03.950354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:03.950354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:03.950454 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:03.950690 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:03.950694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:03.950809 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 22:18:03.950896 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 22:18:03.951018 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":120,"threads_started":1}
I20260923 22:18:03.951851 15635 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39799
I20260923 22:18:03.951969 15635 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:03.952159 15635 heartbeater.cc:511] Master 127.4.207.62:39799 requested a full tablet report, sending...
I20260923 22:18:03.952559 15176 ts_manager.cc:194] Registered new tserver with Master: ba1be0fecd72461e879559048fd71575 (127.4.207.3:37099)
I20260923 22:18:03.952936 15176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:38981
I20260923 22:18:03.956995 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:18:03.964525 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.958323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.4.207.62:39799)
0923 22:18:03.958396 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:03.958398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:03.958522 (+ 124us) server_negotiation.cc:407] Connection header received
0923 22:18:03.958560 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:03.958561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:03.958569 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:03.958620 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:03.959160 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.959768 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.960496 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.960675 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.960724 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.961913 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:03.961914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:03.961915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:03.961921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:03.963410 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.963765 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.963765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.963766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.963828 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:03.964025 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:03.964026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:03.964027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:03.964212 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:03.964338 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:03.964388 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 22:18:03.964457 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 22:18:03.966152 15176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:18:03.967888 15176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:18:03.974213 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.970293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:37099 (local address 127.0.0.1:57256)
0923 22:18:03.970395 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:03.970405 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:03.970458 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:03.970991 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:03.970992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:03.970994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:03.971046 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.971047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.972415 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.972416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.973191 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.973194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.973893 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.973894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.973979 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.973993 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 22:18:03.974078 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 22:18:03.974130 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":45,"threads_started":1}
I20260923 22:18:03.974226 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.970594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57256 (local address 127.4.207.3:37099)
0923 22:18:03.970811 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:03.970815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:03.970825 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:18:03.970863 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:03.970865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:03.970880 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:03.971115 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:03.971182 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.972304 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.973285 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.973814 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.973893 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.973964 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:03.974105 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 22:18:03.974144 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260923 22:18:03.974754 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.970420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:38521 (local address 127.0.0.1:52954)
0923 22:18:03.970704 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:03.970714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:03.970784 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:03.971054 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:03.971055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:03.971057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:03.971114 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.971116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.972092 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.972093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.973479 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.973482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.974518 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.974519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.974584 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.974587 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:03.974636 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:18:03.974678 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":52,"thread_start_us":47,"threads_started":1}
I20260923 22:18:03.974864 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.970071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:43075 (local address 127.0.0.1:36286)
0923 22:18:03.970198 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:03.970240 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:03.970321 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:03.970645 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:03.970649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:03.970651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:03.970722 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.970725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.972142 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.972142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.973734 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:03.973736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.974633 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.974634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:03.974699 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.974702 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:03.974752 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:18:03.974797 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":70,"thread_start_us":80,"threads_started":1}
I20260923 22:18:03.975057 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.970548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.4.207.1:38521)
0923 22:18:03.970691 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:03.970695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:03.970841 (+ 146us) server_negotiation.cc:407] Connection header received
0923 22:18:03.970886 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:03.970888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:03.970904 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:03.970974 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:03.971174 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.971986 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.973572 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.974345 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.974418 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.974482 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:03.974925 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 22:18:03.974967 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260923 22:18:03.975142 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:03.970175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.4.207.2:43075)
0923 22:18:03.970358 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:03.970363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:03.970375 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:18:03.970421 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:03.970423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:03.970466 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:03.970542 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:03.970878 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.972054 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.973826 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:03.974560 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:03.974631 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:03.974683 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:03.975021 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 22:18:03.975064 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260923 22:18:03.975196 15569 tablet_service.cc:1467] Processing CreateTablet for tablet e60456ad937949618f58eabeb8f66013 (DEFAULT_TABLE table=test-table [id=f20c7fc554b84e5ab9035963ed7ff293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:03.975603 15569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60456ad937949618f58eabeb8f66013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:03.975733 15301 tablet_service.cc:1467] Processing CreateTablet for tablet e60456ad937949618f58eabeb8f66013 (DEFAULT_TABLE table=test-table [id=f20c7fc554b84e5ab9035963ed7ff293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:03.976085 15301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60456ad937949618f58eabeb8f66013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:03.978432 15649 tablet_bootstrap.cc:492] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: Bootstrap starting.
I20260923 22:18:03.978770 15650 tablet_bootstrap.cc:492] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451: Bootstrap starting.
I20260923 22:18:03.978812 15435 tablet_service.cc:1467] Processing CreateTablet for tablet e60456ad937949618f58eabeb8f66013 (DEFAULT_TABLE table=test-table [id=f20c7fc554b84e5ab9035963ed7ff293]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:03.979167 15649 tablet_bootstrap.cc:654] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:03.979161 15435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60456ad937949618f58eabeb8f66013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:03.979527 15650 tablet_bootstrap.cc:654] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:03.979825 15649 log.cc:824] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:03.980083 15650 log.cc:824] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:03.980561 15649 tablet_bootstrap.cc:492] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: No bootstrap required, opened a new log
I20260923 22:18:03.980621 15649 ts_tablet_manager.cc:1397] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:18:03.982251 15653 tablet_bootstrap.cc:492] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Bootstrap starting.
I20260923 22:18:03.982362 15649 raft_consensus.cc:348] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:03.982544 15649 raft_consensus.cc:374] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:03.982589 15649 raft_consensus.cc:729] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba1be0fecd72461e879559048fd71575, State: Initialized, Role: FOLLOWER
I20260923 22:18:03.982708 15649 consensus_queue.cc:260] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:03.982856 15650 tablet_bootstrap.cc:492] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451: No bootstrap required, opened a new log
I20260923 22:18:03.982895 15653 tablet_bootstrap.cc:654] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:03.982933 15650 ts_tablet_manager.cc:1397] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 22:18:03.983019 15649 ts_tablet_manager.cc:1428] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:18:03.983147 15635 heartbeater.cc:503] Master 127.4.207.62:39799 was elected leader, sending a full tablet report...
I20260923 22:18:03.983491 15653 log.cc:824] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:03.984536 15653 tablet_bootstrap.cc:492] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: No bootstrap required, opened a new log
I20260923 22:18:03.984608 15653 ts_tablet_manager.cc:1397] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:18:03.984962 15650 raft_consensus.cc:348] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:03.985134 15650 raft_consensus.cc:374] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:03.985168 15650 raft_consensus.cc:729] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130aa307052b410ba7b200a08f25b451, State: Initialized, Role: FOLLOWER
I20260923 22:18:03.985288 15650 consensus_queue.cc:260] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:03.985576 15650 ts_tablet_manager.cc:1428] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:18:03.985599 15367 heartbeater.cc:503] Master 127.4.207.62:39799 was elected leader, sending a full tablet report...
I20260923 22:18:03.986733 15653 raft_consensus.cc:348] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:03.986866 15653 raft_consensus.cc:374] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:03.986887 15653 raft_consensus.cc:729] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c703d99feb4872aee2b3fea586ee78, State: Initialized, Role: FOLLOWER
I20260923 22:18:03.986974 15653 consensus_queue.cc:260] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:03.987272 15653 ts_tablet_manager.cc:1428] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:18:03.987285 15501 heartbeater.cc:503] Master 127.4.207.62:39799 was elected leader, sending a full tablet report...
I20260923 22:18:04.006996 15654 raft_consensus.cc:482] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:18:04.007136 15654 raft_consensus.cc:504] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:04.007598 15654 leader_election.cc:288] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 34c703d99feb4872aee2b3fea586ee78 (127.4.207.2:43075), 130aa307052b410ba7b200a08f25b451 (127.4.207.1:38521)
I20260923 22:18:04.010897 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:04.007770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:43075 (local address 127.4.207.3:47157)
0923 22:18:04.007827 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:04.007840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:04.007909 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:04.008221 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:04.008222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:04.008224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:04.008303 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:04.008306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.009048 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.009049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:04.009971 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:04.009973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.010592 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.010593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:04.010679 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:04.010684 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:18:04.010738 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 22:18:04.010793 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 22:18:04.010898 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:04.008009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:47157 (local address 127.4.207.2:43075)
0923 22:18:04.008065 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:04.008067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:04.008076 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:18:04.008104 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:04.008105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:04.008117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:04.008186 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:04.008401 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.008976 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.010053 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.010518 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.010574 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:04.010583 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:04.010756 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 22:18:04.010792 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:18:04.011021 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:04.008013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:34257 (local address 127.4.207.1:38521)
0923 22:18:04.008081 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:04.008082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:04.008200 (+ 118us) server_negotiation.cc:407] Connection header received
0923 22:18:04.008249 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:04.008250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:04.008264 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:04.008334 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:04.008551 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.009107 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.010170 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.010642 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.010717 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:04.010729 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:04.010899 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 22:18:04.010945 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:18:04.011246 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:04.007900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:38521 (local address 127.4.207.3:34257)
0923 22:18:04.008133 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:04.008143 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:04.008194 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:04.008401 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:04.008402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:04.008404 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:04.008471 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:04.008472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.009217 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.009218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:04.010085 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:04.010087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.010749 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.010750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:04.010817 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:04.010821 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:18:04.010878 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 22:18:04.010934 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
I20260923 22:18:04.011355 15455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60456ad937949618f58eabeb8f66013" candidate_uuid: "ba1be0fecd72461e879559048fd71575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c703d99feb4872aee2b3fea586ee78" is_pre_election: true
I20260923 22:18:04.011570 15455 raft_consensus.cc:2454] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba1be0fecd72461e879559048fd71575 in term 0.
I20260923 22:18:04.011699 15321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60456ad937949618f58eabeb8f66013" candidate_uuid: "ba1be0fecd72461e879559048fd71575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130aa307052b410ba7b200a08f25b451" is_pre_election: true
I20260923 22:18:04.011894 15321 raft_consensus.cc:2454] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba1be0fecd72461e879559048fd71575 in term 0.
I20260923 22:18:04.011874 15523 leader_election.cc:302] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34c703d99feb4872aee2b3fea586ee78, ba1be0fecd72461e879559048fd71575; no voters:
I20260923 22:18:04.012066 15654 raft_consensus.cc:2781] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:18:04.012137 15654 raft_consensus.cc:482] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:18:04.012166 15654 raft_consensus.cc:3037] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:04.012878 15654 raft_consensus.cc:504] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:04.013015 15654 leader_election.cc:288] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [CANDIDATE]: Term 1 election: Requested vote from peers 34c703d99feb4872aee2b3fea586ee78 (127.4.207.2:43075), 130aa307052b410ba7b200a08f25b451 (127.4.207.1:38521)
I20260923 22:18:04.013172 15455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60456ad937949618f58eabeb8f66013" candidate_uuid: "ba1be0fecd72461e879559048fd71575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c703d99feb4872aee2b3fea586ee78"
I20260923 22:18:04.013221 15321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60456ad937949618f58eabeb8f66013" candidate_uuid: "ba1be0fecd72461e879559048fd71575" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130aa307052b410ba7b200a08f25b451"
I20260923 22:18:04.013275 15455 raft_consensus.cc:3037] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:04.013312 15321 raft_consensus.cc:3037] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:04.014034 15455 raft_consensus.cc:2454] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba1be0fecd72461e879559048fd71575 in term 1.
I20260923 22:18:04.014037 15321 raft_consensus.cc:2454] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba1be0fecd72461e879559048fd71575 in term 1.
I20260923 22:18:04.014228 15523 leader_election.cc:302] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130aa307052b410ba7b200a08f25b451, ba1be0fecd72461e879559048fd71575; no voters:
I20260923 22:18:04.014360 15654 raft_consensus.cc:2781] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:18:04.014549 15654 raft_consensus.cc:686] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 1 LEADER]: Becoming Leader. State: Replica: ba1be0fecd72461e879559048fd71575, State: Running, Role: LEADER
I20260923 22:18:04.014657 15654 consensus_queue.cc:237] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } }
I20260923 22:18:04.015590 15176 catalog_manager.cc:5495] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba1be0fecd72461e879559048fd71575 (127.4.207.3). New cstate: current_term: 1 leader_uuid: "ba1be0fecd72461e879559048fd71575" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba1be0fecd72461e879559048fd71575" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 37099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 } health_report { overall_health: UNKNOWN } } }
I20260923 22:18:04.032265 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:04.030029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57264 (local address 127.4.207.3:37099)
0923 22:18:04.030082 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:04.030084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:04.030090 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:18:04.030122 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:04.030123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:04.030140 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:04.030203 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:18:04.030332 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.030792 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.031607 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:04.031792 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:04.031853 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:04.031983 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:18:04.032085 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:18:04.032142 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:04.032171 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 22:18:04.032188 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:18:04.034202 15455 raft_consensus.cc:1261] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 1 FOLLOWER]: Refusing update from remote peer ba1be0fecd72461e879559048fd71575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:18:04.034199 15321 raft_consensus.cc:1261] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 1 FOLLOWER]: Refusing update from remote peer ba1be0fecd72461e879559048fd71575: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:18:04.034554 15654 consensus_queue.cc:1035] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "130aa307052b410ba7b200a08f25b451" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 38521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:18:04.034722 15659 consensus_queue.cc:1035] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c703d99feb4872aee2b3fea586ee78" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 43075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:18:04.035885 15502 tablet.cc:2369] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:18:04.039790 15175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:18:04.039884 15175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:18:04.040689 15175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4785cbd9bcc14024b9d8d7a4a8e889df: Sending DeleteTablet for 3 replicas of tablet e60456ad937949618f58eabeb8f66013
I20260923 22:18:04.041054 15435 tablet_service.cc:1514] Processing DeleteTablet for tablet e60456ad937949618f58eabeb8f66013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:04 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:36286
I20260923 22:18:04.041100 15301 tablet_service.cc:1514] Processing DeleteTablet for tablet e60456ad937949618f58eabeb8f66013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:04 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:52954
I20260923 22:18:04.041273 15569 tablet_service.cc:1514] Processing DeleteTablet for tablet e60456ad937949618f58eabeb8f66013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:04 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:57256
I20260923 22:18:04.041306 15673 tablet_replica.cc:330] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: stopping tablet replica
I20260923 22:18:04.041415 15673 raft_consensus.cc:2229] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:18:04.041476 15675 tablet_replica.cc:330] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: stopping tablet replica
I20260923 22:18:04.041517 15673 raft_consensus.cc:2258] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:18:04.041563 15675 raft_consensus.cc:2229] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:18:04.041754 15674 tablet_replica.cc:330] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451: stopping tablet replica
I20260923 22:18:04.041769 15675 raft_consensus.cc:2258] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:18:04.041908 15674 raft_consensus.cc:2229] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:18:04.041997 15673 ts_tablet_manager.cc:1905] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:18:04.042004 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 15238
I20260923 22:18:04.042114 15674 raft_consensus.cc:2258] T e60456ad937949618f58eabeb8f66013 P 130aa307052b410ba7b200a08f25b451 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:18:04.044751 15673 ts_tablet_manager.cc:1918] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:18:04.044837 15673 log.cc:1196] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/e60456ad937949618f58eabeb8f66013
I20260923 22:18:04.045131 15673 ts_tablet_manager.cc:1939] T e60456ad937949618f58eabeb8f66013 P 34c703d99feb4872aee2b3fea586ee78: Deleting consensus metadata
I20260923 22:18:04.045600 15161 catalog_manager.cc:4841] TS 34c703d99feb4872aee2b3fea586ee78 (127.4.207.2:43075): tablet e60456ad937949618f58eabeb8f66013 (table test-table [id=f20c7fc554b84e5ab9035963ed7ff293]) successfully deleted
W20260923 22:18:04.046507 15161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:38521 (error 108)
I20260923 22:18:04.047734 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 15372
I20260923 22:18:04.047902 15675 ts_tablet_manager.cc:1905] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:18:04.049651 15675 ts_tablet_manager.cc:1918] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:18:04.049751 15675 log.cc:1196] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/e60456ad937949618f58eabeb8f66013
I20260923 22:18:04.049994 15675 ts_tablet_manager.cc:1939] T e60456ad937949618f58eabeb8f66013 P ba1be0fecd72461e879559048fd71575: Deleting consensus metadata
I20260923 22:18:04.050448 15161 catalog_manager.cc:4841] TS ba1be0fecd72461e879559048fd71575 (127.4.207.3:37099): tablet e60456ad937949618f58eabeb8f66013 (table test-table [id=f20c7fc554b84e5ab9035963ed7ff293]) successfully deleted
I20260923 22:18:04.051898 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:04.051760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:38521 (local address 127.0.0.1:52962)
0923 22:18:04.051819 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:04.051866 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:38521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260923 22:18:04.052054 15161 catalog_manager.cc:4584] TS 130aa307052b410ba7b200a08f25b451 (127.4.207.1:38521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e60456ad937949618f58eabeb8f66013: Network error: Client connection negotiation failed: client connection to 127.4.207.1:38521: connect: Connection refused (error 111)
I20260923 22:18:04.054924 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 15506
I20260923 22:18:04.060021 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 15144
2026-09-23T22:18:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1909 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:18:04 dist-test-slave-7511 krb5kdc[15679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:18:04 dist-test-slave-7511 krb5kdc[15679](info): set up 2 sockets
Sep 23 22:18:04 dist-test-slave-7511 krb5kdc[15679](info): commencing operation
krb5kdc: starting...
W20260923 22:18:05.112645 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:18:05 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:18:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:18:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.207.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.207.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/rangeradmin_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/rangeradmin_127.4.207.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.207.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.207.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/rangerlookup_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/rangerlookup_127.4.207.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.207.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.207.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/HTTP_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/HTTP_127.4.207.20@KRBTEST.COM.keytab.
I20260923 22:18:05.282297 4924 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMonjHj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/postgres -l logfile start
2026-09-23 22:18:07.851 UTC [15709] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 22:18:07.851 UTC [15709] LOG: listening on IPv4 address "127.4.207.20", port 34729
2026-09-23 22:18:07.855 UTC [15709] LOG: listening on Unix socket "/tmp/.s.PGSQL.34729"
2026-09-23 22:18:07.890 UTC [15712] LOG: database system was shut down at 2026-09-23 22:18:06 UTC
2026-09-23 22:18:07.894 UTC [15709] LOG: database system is ready to accept connections
I20260923 22:18:08.828480 4924 mini_postgres.cc:96] Postgres bound to 34729
2026-09-23 22:18:08.834 UTC [15720] FATAL: database "slave" does not exist
127.4.207.20:34729 - accepting connections
I20260923 22:18:08.834973 4924 mini_ranger.cc:161] Starting Ranger...
I20260923 22:18:08.850153 4924 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 22:18:09.714277 4924 mini_ranger.cc:87] Created ranger Postgres database
I20260923 22:18:09.714370 4924 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-admin
2026-09-23 22:18:09,999 [I] DB FLAVOR :POSTGRES
2026-09-23 22:18:09,999 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 22:18:09,999 [I] Checking connection..
2026-09-23 22:18:10,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 22:18:10,255 [I] Checking connection passed.
2026-09-23 22:18:10,255 [I] --------- Verifying version history table ---------
2026-09-23 22:18:10,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:10,520 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 22:18:10,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:10,775 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 22:18:10,775 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 22:18:10,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 22:18:11.034 UTC [15784] WARNING: there is no transaction in progress
2026-09-23 22:18:11,041 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 22:18:11,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:11,308 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 22:18:11,308 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 22:18:11,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 22:18:11,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 22:18:11,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7511.c.gcp-upstream.internal','N') ;"
2026-09-23 22:18:12,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 22:18:12,278 [I] Table x_portal_user does not exist in database ranger
2026-09-23 22:18:12,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 22:18:12,531 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 22:18:12,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 22:18:12,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 22:18:13,028 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 22:18:13,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 22:18:14.042 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.061 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.080 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.095 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.112 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.132 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.189 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.196 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.210 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.221 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.232 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.247 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.257 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.264 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.272 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14.280 UTC [15892] WARNING: there is no transaction in progress
2026-09-23 22:18:14,930 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 22:18:14,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 22:18:15,156 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 22:18:15,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 22:18:15,400 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 22:18:15,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 22:18:15,675 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 22:18:15,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 22:18:15,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 22:18:16,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7511.c.gcp-upstream.internal';"
2026-09-23 22:18:16,419 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 22:18:16,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 22:18:16,649 [I] DB_PATCHES have already been applied
I20260923 22:18:16.655560 4924 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMonjHj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 22:18:16.655629 4924 mini_ranger.cc:193] Using host: 127.4.207.20
I20260923 22:18:16.658685 4924 mini_ranger.cc:227] Ranger admin URL: http://127.4.207.20:32949
Sep 23, 2026 10:18:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 10:18:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 10:18:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 10:18:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 10:18:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 10:18:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 10:18:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 22:18:17.750048 4924 mini_ranger.cc:160] Time spent starting Ranger: real 8.915s user 0.000s sys 0.009s
Sep 23, 2026 10:18:37 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 10:18:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 23, 2026 10:18:37 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 10:18:37 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 10:18:38 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 22:18:38.880640 4924 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.207.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.207.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/rangerkms_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/rangerkms_127.4.207.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.207.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.207.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.207.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/HTTP_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.207.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/HTTP_127.4.207.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 22:18:38 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201918, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 22:18:38.966637 4924 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 22:18:38.976354 4924 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 22:18:39.842759 4924 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 22:18:39.842873 4924 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms
I20260923 22:18:39.842896 4924 mini_ranger_kms.cc:227] Using postgres at 127.4.207.20:34729
2026-09-23 22:18:40,065 [I] DB FLAVOR :POSTGRES
2026-09-23 22:18:40,066 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 22:18:40,066 [I] Checking connection
2026-09-23 22:18:40,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 22:18:40,367 [I] connection success
2026-09-23 22:18:40,367 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 22:18:40,367 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 22:18:40,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 22:18:40,614 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 22:18:40,615 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 22:18:40,615 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 22:18:40,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:34729/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 22:18:40,869 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 22:18:41.293982 4924 mini_ranger_kms.cc:310] Created kms service
I20260923 22:18:41.485277 4924 mini_ranger_kms.cc:326] Created kudu user
I20260923 22:18:41.525574 4924 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 22:18:41 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201918, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:18:41.884958 4924 mini_ranger_kms.cc:384] Added ranger policy
I20260923 22:18:41.885090 4924 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 22:18:41.885113 4924 mini_ranger_kms.cc:240] Using host: 127.4.207.20
I20260923 22:18:41.888994 4924 mini_ranger_kms.cc:276] Ranger KMS PID: 16072
I20260923 22:18:41.889124 4924 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.207.20:36421
Sep 23, 2026 10:18:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 10:18:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 10:18:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 10:18:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 10:18:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 10:18:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 23 22:18:44 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201924, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.207.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:44.146840 4924 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.180s user 0.015s sys 0.186s
I20260923 22:18:44.146972 4924 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 22:18:44.147051 4924 mini_ranger_kms.cc:401] 127.4.207.20:36421/kms/v1/keys
Sep 23 22:18:44 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201924, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.207.20@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
Sep 23 22:18:44 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201924, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.207.20@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
Sep 23 22:18:47 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:18:47.062093 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:40489
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:40489
--ranger_config_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:36421/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:47.149422 16123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:47.149631 16123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:47.149652 16123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:47.150780 16123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:18:47.150815 16123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:47.150831 16123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:47.150846 16123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:18:47.150857 16123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:18:47.152392 16123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.207.20:36421/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:40489
--ranger_config_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:47.152580 16123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:47.152743 16123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:47.158311 16131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:47.158311 16129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:47.158375 16123 server_base.cc:1034] running on GCE node
W20260923 22:18:47.158401 16128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:47.158897 16123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:47.159215 16123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:47.160356 16123 hybrid_clock.cc:648] HybridClock initialized: now 1790201927160320 us; error 48 us; skew 500 ppm
Sep 23 22:18:47 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201927, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:47.162894 16123 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:18:47.163345 16123 webserver.cc:460] Webserver started at http://127.4.207.62:33803/ using document root <none> and password file <none>
I20260923 22:18:47.163618 16123 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:47.163679 16123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:47.163825 16123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:18:47 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201927, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:18:47.227977 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "487b48cca5784e19afda48f41cba430e"
format_stamp: "Formatted at 2026-09-23 22:18:47 on dist-test-slave-7511"
server_key: "38f45225a4aba54a9beb8cbb782c7db5"
server_key_iv: "b4d42d7569c98ca751aeb6937e90a625"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:47.228484 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "487b48cca5784e19afda48f41cba430e"
format_stamp: "Formatted at 2026-09-23 22:18:47 on dist-test-slave-7511"
server_key: "38f45225a4aba54a9beb8cbb782c7db5"
server_key_iv: "b4d42d7569c98ca751aeb6937e90a625"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:47.230361 16123 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 22:18:47.245178 16138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:47.245450 16123 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:18:47.245564 16123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "487b48cca5784e19afda48f41cba430e"
format_stamp: "Formatted at 2026-09-23 22:18:47 on dist-test-slave-7511"
server_key: "38f45225a4aba54a9beb8cbb782c7db5"
server_key_iv: "b4d42d7569c98ca751aeb6937e90a625"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:47.245666 16123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:47.280602 16123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:47.282872 16123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:47.283048 16123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:47.287802 16123 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:40489
I20260923 22:18:47.287884 16190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:40489 every 8 connection(s)
I20260923 22:18:47.288259 16123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:18:47.289157 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 16123
I20260923 22:18:47.289258 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:18:47.289453 4924 external_mini_cluster.cc:1428] Setting key 12de780f8e818f60b1c1a6915206579f
I20260923 22:18:47.289924 16191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:47.293403 16191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e: Bootstrap starting.
I20260923 22:18:47.294142 16191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:47.294457 16191 log.cc:824] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:47.298475 16191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e: No bootstrap required, opened a new log
I20260923 22:18:47.300541 16191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b48cca5784e19afda48f41cba430e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40489 } }
I20260923 22:18:47.300738 16191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:47.300776 16191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 487b48cca5784e19afda48f41cba430e, State: Initialized, Role: FOLLOWER
I20260923 22:18:47.300952 16191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b48cca5784e19afda48f41cba430e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40489 } }
I20260923 22:18:47.301038 16191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:18:47.301069 16191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:18:47.301116 16191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:47.301892 16191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b48cca5784e19afda48f41cba430e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40489 } }
I20260923 22:18:47.302012 16191 leader_election.cc:302] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 487b48cca5784e19afda48f41cba430e; no voters:
I20260923 22:18:47.302186 16191 leader_election.cc:288] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:18:47.302318 16196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:18:47.302474 16191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:18:47.302500 16196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [term 1 LEADER]: Becoming Leader. State: Replica: 487b48cca5784e19afda48f41cba430e, State: Running, Role: LEADER
I20260923 22:18:47.302735 16196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b48cca5784e19afda48f41cba430e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40489 } }
Sep 23 22:18:47 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:18:47.303174 16196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "487b48cca5784e19afda48f41cba430e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b48cca5784e19afda48f41cba430e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40489 } } }
I20260923 22:18:47.303532 16196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [sys.catalog]: This master's current role is: LEADER
I20260923 22:18:47.303737 16196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 487b48cca5784e19afda48f41cba430e. Latest consensus state: current_term: 1 leader_uuid: "487b48cca5784e19afda48f41cba430e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "487b48cca5784e19afda48f41cba430e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40489 } } }
I20260923 22:18:47.304172 16196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e [sys.catalog]: This master's current role is: LEADER
I20260923 22:18:47.304348 16210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:18:47.303996 16191 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 22:18:47.305234 16210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:18:47.307327 16210 catalog_manager.cc:1347] Generated new cluster ID: 8f66129eee204c789c783f46502aa540
I20260923 22:18:47.307384 16210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:18:47.309016 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:47.290540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.4.207.62:40489)
0923 22:18:47.295633 (+ 5093us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:47.295645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:47.295688 (+ 43us) server_negotiation.cc:407] Connection header received
0923 22:18:47.296453 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:47.296464 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:47.296554 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:47.296748 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:47.297419 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:47.298686 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:47.299528 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:47.300363 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:47.300495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:47.303567 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:47.303571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:47.303571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:47.303585 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:47.305605 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:47.306120 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:47.306123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:47.306124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:47.306225 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:47.306477 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:47.306478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:47.306478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:47.306773 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:47.306911 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:47.307194 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 22:18:47.307321 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5015,"thread_start_us":104,"threads_started":1}
I20260923 22:18:47.320173 16210 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:18:47.321038 16210 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:18:47.330231 16210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e: Generated new TSK 0
I20260923 22:18:47.330461 16210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:18:48.030884 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:36421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:40489
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:48.139353 16236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:48.139663 16236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:48.139705 16236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:48.140937 16236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:48.141004 16236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:48.141049 16236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:18:48.142585 16236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.207.20:36421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:48.142858 16236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:48.143134 16236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:48.149428 16242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:48.149602 16244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:48.149432 16241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:48.150162 16236 server_base.cc:1034] running on GCE node
I20260923 22:18:48.150456 16236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:48.150784 16236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:48.152060 16236 hybrid_clock.cc:648] HybridClock initialized: now 1790201928152039 us; error 33 us; skew 500 ppm
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:48.155846 16236 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:18:48.156448 16236 webserver.cc:460] Webserver started at http://127.4.207.1:41161/ using document root <none> and password file <none>
I20260923 22:18:48.156741 16236 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:48.156810 16236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:48.156944 16236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:18:48.169812 16236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "c668153b01e24df9a40beb9dd38822ed"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "8298d7ef57a1220d04f0f3c3d4add14d"
server_key_iv: "d14b68622cfa7c9ea87d2adc06e44378"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.170320 16236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "c668153b01e24df9a40beb9dd38822ed"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "8298d7ef57a1220d04f0f3c3d4add14d"
server_key_iv: "d14b68622cfa7c9ea87d2adc06e44378"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.175704 16236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 22:18:48.194291 16253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.194507 16236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260923 22:18:48.194636 16236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "c668153b01e24df9a40beb9dd38822ed"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "8298d7ef57a1220d04f0f3c3d4add14d"
server_key_iv: "d14b68622cfa7c9ea87d2adc06e44378"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.194744 16236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:48.218082 16236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:48.220567 16236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:48.220719 16236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:48.220979 16236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:48.221340 16236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:48.221386 16236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.221421 16236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:48.221446 16236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.228627 16236 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:40475
I20260923 22:18:48.229086 16236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:18:48.232074 16366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:40475 every 8 connection(s)
I20260923 22:18:48.235931 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 16236
I20260923 22:18:48.236045 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:18:48.236255 4924 external_mini_cluster.cc:1428] Setting key a8b2fdc57d8b08272edad9e9fe87fb67
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:18:48.245260 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.229950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:56409 (local address 127.4.207.62:40489)
0923 22:18:48.234878 (+ 4928us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.234881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.234893 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:18:48.234923 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.234924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.234939 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.235212 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:48.236120 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.237117 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.238655 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.238962 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.239010 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.241705 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:48.241707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:48.241707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:48.241715 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:48.243650 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:48.244149 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:48.244151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:48.244151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:48.244215 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:48.244500 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:48.244501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:48.244502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:48.244636 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:48.244687 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.245055 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.245146 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4888,"thread_start_us":90,"threads_started":1}
I20260923 22:18:48.245848 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.229919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:40489 (local address 127.4.207.1:56409)
0923 22:18:48.231598 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.231653 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.232416 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.235326 (+ 2910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:48.235330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:48.235696 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:48.235993 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.236000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.237231 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.237232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.238543 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.238546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.239116 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.239117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.239260 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.239913 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:48.239924 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:48.241588 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:48.243751 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:48.243758 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:48.243760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:48.244056 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:48.244280 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:48.244281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:48.244281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:48.244425 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:48.244717 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:48.244722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:48.244902 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 22:18:48.245416 (+ 514us) client_negotiation.cc:240] Negotiation successful
0923 22:18:48.245594 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1626,"thread_start_us":83,"threads_started":1}
I20260923 22:18:48.246397 16367 heartbeater.cc:347] Connected to a master server at 127.4.207.62:40489
I20260923 22:18:48.246505 16367 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
I20260923 22:18:48.246721 16367 heartbeater.cc:511] Master 127.4.207.62:40489 requested a full tablet report, sending...
I20260923 22:18:48.247270 16155 ts_manager.cc:194] Registered new tserver with Master: c668153b01e24df9a40beb9dd38822ed (127.4.207.1:40475)
I20260923 22:18:48.248257 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:56409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:18:48.270744 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:36421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:40489
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 22:18:48.395251 16373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:48.395583 16373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:48.395622 16373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:48.397409 16373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:48.397473 16373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:48.397514 16373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:18:48.400357 16373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.207.20:36421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:48.400668 16373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:48.400967 16373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:48.407735 16380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:48.407753 16381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:48.408113 16383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:48.408390 16373 server_base.cc:1034] running on GCE node
I20260923 22:18:48.408641 16373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:48.408967 16373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:48.410159 16373 hybrid_clock.cc:648] HybridClock initialized: now 1790201928410108 us; error 66 us; skew 500 ppm
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:48.413604 16373 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:18:48.414170 16373 webserver.cc:460] Webserver started at http://127.4.207.2:41429/ using document root <none> and password file <none>
I20260923 22:18:48.414438 16373 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:48.414501 16373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:48.414634 16373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:18:48.431890 16373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "e924c06d7234431f9640a262b584bd1b"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "4d17951a1c0ac18e7fef2d270eed162b"
server_key_iv: "dded46720cb021f2df16d07e25fa1198"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.432536 16373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "e924c06d7234431f9640a262b584bd1b"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "4d17951a1c0ac18e7fef2d270eed162b"
server_key_iv: "dded46720cb021f2df16d07e25fa1198"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.435850 16373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260923 22:18:48.455653 16391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.455947 16373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260923 22:18:48.456048 16373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "e924c06d7234431f9640a262b584bd1b"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "4d17951a1c0ac18e7fef2d270eed162b"
server_key_iv: "dded46720cb021f2df16d07e25fa1198"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.456149 16373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:48.481487 16373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:48.483893 16373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:48.484057 16373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:48.484297 16373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:48.484704 16373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:48.484757 16373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.484797 16373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:48.484838 16373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.492533 16373 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:35475
I20260923 22:18:48.492961 16373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:18:48.494236 16504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:35475 every 8 connection(s)
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:18:48.500300 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 16373
I20260923 22:18:48.500433 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:18:48.500663 4924 external_mini_cluster.cc:1428] Setting key 673dbf303620eba455c5070d24c73c01
I20260923 22:18:48.504055 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.494096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:42859 (local address 127.4.207.62:40489)
0923 22:18:48.494138 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.494140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.494150 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:18:48.494905 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.494906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.494919 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.494974 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:48.495797 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.496338 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.497564 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.497795 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.497874 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.500309 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:48.500310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:48.500311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:48.500319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:48.502249 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:48.502816 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:48.502817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:48.502818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:48.502877 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:48.503172 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:48.503173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:48.503174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:48.503333 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:48.503389 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.503829 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.503930 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:18:48.504932 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.493765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:40489 (local address 127.4.207.2:42859)
0923 22:18:48.493963 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.494009 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.494744 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.495057 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:48.495062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:48.495259 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:48.495534 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.495540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.496462 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.496463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.497473 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.497476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.497882 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.497883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.498018 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.498623 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:48.498634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:48.500187 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:48.502350 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:48.502368 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:48.502371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:48.502711 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:48.502949 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:48.502949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:48.502950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:48.503092 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:48.503460 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:48.503467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:48.503670 (+ 203us) client_negotiation.cc:769] Sending connection context
0923 22:18:48.504448 (+ 778us) client_negotiation.cc:240] Negotiation successful
0923 22:18:48.504635 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":105,"threads_started":1}
I20260923 22:18:48.505724 16505 heartbeater.cc:347] Connected to a master server at 127.4.207.62:40489
I20260923 22:18:48.505851 16505 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:48.506045 16505 heartbeater.cc:511] Master 127.4.207.62:40489 requested a full tablet report, sending...
I20260923 22:18:48.506443 16155 ts_manager.cc:194] Registered new tserver with Master: e924c06d7234431f9640a262b584bd1b (127.4.207.2:35475)
I20260923 22:18:48.506982 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:42859
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:18:48.530993 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:36421/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:40489
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:18:48.628222 16510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:18:48.628476 16510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:18:48.628515 16510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:18:48.629685 16510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:18:48.629755 16510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:18:48.629798 16510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:18:48.631263 16510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.207.20:36421/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:40489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:18:48.631551 16510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:18:48.631793 16510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:18:48.636844 16516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:48.636933 16518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:18:48.636854 16515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:18:48.637229 16510 server_base.cc:1034] running on GCE node
I20260923 22:18:48.637451 16510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:18:48.637703 16510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:18:48.638875 16510 hybrid_clock.cc:648] HybridClock initialized: now 1790201928638861 us; error 29 us; skew 500 ppm
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:18:48.641404 16510 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:18:48.641846 16510 webserver.cc:460] Webserver started at http://127.4.207.3:33241/ using document root <none> and password file <none>
I20260923 22:18:48.642095 16510 fs_manager.cc:359] Metadata directory not provided
I20260923 22:18:48.642162 16510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:18:48.642309 16510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:18:48.662317 16510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "8f2ab47f5a184b1c97f64bc669f0a43a"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "90fc1fb74aec69728e05a4e0e9a3fc38"
server_key_iv: "959d82a1aaac8689ed58b553f04f1ba3"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.662778 16510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "8f2ab47f5a184b1c97f64bc669f0a43a"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "90fc1fb74aec69728e05a4e0e9a3fc38"
server_key_iv: "959d82a1aaac8689ed58b553f04f1ba3"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.664657 16510 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260923 22:18:48.679613 16525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.679850 16510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260923 22:18:48.679960 16510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "8f2ab47f5a184b1c97f64bc669f0a43a"
format_stamp: "Formatted at 2026-09-23 22:18:48 on dist-test-slave-7511"
server_key: "90fc1fb74aec69728e05a4e0e9a3fc38"
server_key_iv: "959d82a1aaac8689ed58b553f04f1ba3"
server_key_version: "kuduclusterkey@0"
I20260923 22:18:48.680052 16510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:18:48.705174 16510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:18:48.707907 16510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:18:48.708134 16510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:18:48.708369 16510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:18:48.714208 16510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:18:48.714293 16510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.714373 16510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:18:48.714403 16510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:18:48.722704 16510 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:36021
I20260923 22:18:48.725378 16510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:18:48.728808 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 16510
I20260923 22:18:48.728302 16638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:36021 every 8 connection(s)
I20260923 22:18:48.728937 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:18:48.729140 4924 external_mini_cluster.cc:1428] Setting key bad6359d60c64358a42f8ecac389d612
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:18:48.739960 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.729311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:43121 (local address 127.4.207.62:40489)
0923 22:18:48.729370 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.729373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.729466 (+ 93us) server_negotiation.cc:407] Connection header received
0923 22:18:48.730170 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.730171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.730184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.730216 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:48.731079 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.731907 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.733454 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.733734 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.733781 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.736534 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:48.736535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:48.736536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:48.736544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:48.738273 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:48.738785 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:48.738787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:48.738787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:48.738855 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:48.739144 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:48.739145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:48.739145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:48.739277 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:48.739320 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.739790 (+ 470us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.739869 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:18:48.740733 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.729011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:40489 (local address 127.4.207.3:43121)
0923 22:18:48.729221 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.729277 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.730038 (+ 761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.730434 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:48.730438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:48.730685 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:18:48.730969 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.730975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.732025 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.732026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.733316 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.733319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.733878 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.733879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.734021 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.734646 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:18:48.734657 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:18:48.736420 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:18:48.738371 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:48.738377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:48.738379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:48.738694 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:48.738929 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:18:48.738930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:18:48.738930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:18:48.739063 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:18:48.739375 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:18:48.739381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:18:48.739616 (+ 235us) client_negotiation.cc:769] Sending connection context
0923 22:18:48.740324 (+ 708us) client_negotiation.cc:240] Negotiation successful
0923 22:18:48.740502 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
I20260923 22:18:48.741454 16639 heartbeater.cc:347] Connected to a master server at 127.4.207.62:40489
I20260923 22:18:48.741547 16639 heartbeater.cc:464] Registering TS with master...
I20260923 22:18:48.741735 16639 heartbeater.cc:511] Master 127.4.207.62:40489 requested a full tablet report, sending...
I20260923 22:18:48.742146 16155 ts_manager.cc:194] Registered new tserver with Master: 8f2ab47f5a184b1c97f64bc669f0a43a (127.4.207.3:36021)
I20260923 22:18:48.742659 16155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:43121
I20260923 22:18:48.751495 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:18:48.763973 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.755771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.4.207.62:40489)
0923 22:18:48.755817 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.755820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.756180 (+ 360us) server_negotiation.cc:407] Connection header received
0923 22:18:48.756285 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.756286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.756297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.756328 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:18:48.757197 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.757976 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.759078 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.759379 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.759420 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.761040 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:18:48.761041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:18:48.761041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:18:48.761049 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:18:48.762748 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:48.763139 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:48.763141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:48.763141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:48.763190 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:18:48.763365 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:18:48.763365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:18:48.763365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:18:48.763516 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:18:48.763590 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.763791 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.763878 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:18:48.766207 16155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:18:48.768427 16155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:18:48.778954 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.771218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:36021 (local address 127.0.0.1:39954)
0923 22:18:48.771550 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.771591 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.771934 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.774321 (+ 2387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:48.774324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:48.774326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:48.774406 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.774408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.775803 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.775804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.777463 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.777466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.778405 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.778406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.778487 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.778502 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 22:18:48.778565 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 22:18:48.778859 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":81,"threads_started":1}
I20260923 22:18:48.784570 16652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.771491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:35475 (local address 127.0.0.1:46086)
0923 22:18:48.779328 (+ 7837us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.779337 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.779373 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.780673 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:48.780674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:48.780676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:48.780707 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.780708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.781908 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.781909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.783214 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.783217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.784344 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.784345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.784405 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.784408 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:48.784456 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:18:48.784494 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7825,"thread_start_us":68,"threads_started":1}
I20260923 22:18:48.785271 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.771624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.4.207.3:36021)
0923 22:18:48.774014 (+ 2390us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.774018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.774027 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:18:48.774071 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.774073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.774090 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.774472 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:48.774538 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.775706 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.777593 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.778315 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.784983 (+ 6668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.785078 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.785140 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.785197 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2357,"thread_start_us":70,"threads_started":1}
I20260923 22:18:48.785604 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.771871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.4.207.2:35475)
0923 22:18:48.773929 (+ 2058us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.773935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.779432 (+ 5497us) server_negotiation.cc:407] Connection header received
0923 22:18:48.779506 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.779508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.779524 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.780583 (+ 1059us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:48.780776 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.781816 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.783323 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.784176 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.784239 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.784313 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.785467 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.785510 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2018,"thread_start_us":77,"threads_started":1}
I20260923 22:18:48.786432 16573 tablet_service.cc:1467] Processing CreateTablet for tablet 08501a710c2b45a39ecf86d8430adf51 (DEFAULT_TABLE table=test-table [id=66c184ed71ca43169f01c91b4bf1e012]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:48.786801 16573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08501a710c2b45a39ecf86d8430adf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:48.791872 16430 tablet_service.cc:1467] Processing CreateTablet for tablet 08501a710c2b45a39ecf86d8430adf51 (DEFAULT_TABLE table=test-table [id=66c184ed71ca43169f01c91b4bf1e012]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:48.792222 16430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08501a710c2b45a39ecf86d8430adf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:48.795754 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.771224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:40475 (local address 127.0.0.1:39152)
0923 22:18:48.774194 (+ 2970us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.774204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.774242 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.791739 (+ 17497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:48.791740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:48.791743 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:48.791791 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.791793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.793237 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.793238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.794402 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.794405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.795341 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.795342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.795596 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.795600 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:18:48.795633 (+ 33us) client_negotiation.cc:240] Negotiation successful
0923 22:18:48.795674 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2953,"thread_start_us":97,"threads_started":1}
I20260923 22:18:48.796562 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.771313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.4.207.1:40475)
0923 22:18:48.791525 (+ 20212us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.791530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.791541 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:18:48.791576 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.791578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.791596 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.791857 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:48.791929 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.793120 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.794502 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.795248 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.796349 (+ 1101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.796421 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.796446 (+ 25us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.796491 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20173,"thread_start_us":50,"threads_started":1}
I20260923 22:18:48.797317 16301 tablet_service.cc:1467] Processing CreateTablet for tablet 08501a710c2b45a39ecf86d8430adf51 (DEFAULT_TABLE table=test-table [id=66c184ed71ca43169f01c91b4bf1e012]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:18:48.797640 16301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08501a710c2b45a39ecf86d8430adf51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:18:48.797613 16658 tablet_bootstrap.cc:492] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: Bootstrap starting.
I20260923 22:18:48.799333 16658 tablet_bootstrap.cc:654] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:48.799696 16658 log.cc:824] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:48.801052 16661 tablet_bootstrap.cc:492] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed: Bootstrap starting.
I20260923 22:18:48.801770 16661 tablet_bootstrap.cc:654] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:48.802126 16661 log.cc:824] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:48.803227 16661 tablet_bootstrap.cc:492] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed: No bootstrap required, opened a new log
I20260923 22:18:48.803289 16661 ts_tablet_manager.cc:1397] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:18:48.805725 16661 raft_consensus.cc:348] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.805891 16661 raft_consensus.cc:374] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:48.805919 16661 raft_consensus.cc:729] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c668153b01e24df9a40beb9dd38822ed, State: Initialized, Role: FOLLOWER
I20260923 22:18:48.806154 16659 tablet_bootstrap.cc:492] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b: Bootstrap starting.
I20260923 22:18:48.806200 16661 consensus_queue.cc:260] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.806581 16661 ts_tablet_manager.cc:1428] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:18:48.806733 16367 heartbeater.cc:503] Master 127.4.207.62:40489 was elected leader, sending a full tablet report...
I20260923 22:18:48.807225 16659 tablet_bootstrap.cc:654] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b: Neither blocks nor log segments found. Creating new log.
I20260923 22:18:48.807560 16659 log.cc:824] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:18:48.808369 16658 tablet_bootstrap.cc:492] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: No bootstrap required, opened a new log
I20260923 22:18:48.808442 16658 ts_tablet_manager.cc:1397] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260923 22:18:48.810731 16659 tablet_bootstrap.cc:492] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b: No bootstrap required, opened a new log
I20260923 22:18:48.810858 16659 ts_tablet_manager.cc:1397] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260923 22:18:48.810844 16658 raft_consensus.cc:348] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.811019 16658 raft_consensus.cc:374] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:48.811062 16658 raft_consensus.cc:729] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2ab47f5a184b1c97f64bc669f0a43a, State: Initialized, Role: FOLLOWER
I20260923 22:18:48.811216 16658 consensus_queue.cc:260] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.811941 16658 ts_tablet_manager.cc:1428] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260923 22:18:48.812062 16639 heartbeater.cc:503] Master 127.4.207.62:40489 was elected leader, sending a full tablet report...
I20260923 22:18:48.813190 16659 raft_consensus.cc:348] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.813350 16659 raft_consensus.cc:374] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:18:48.813380 16659 raft_consensus.cc:729] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e924c06d7234431f9640a262b584bd1b, State: Initialized, Role: FOLLOWER
I20260923 22:18:48.813493 16659 consensus_queue.cc:260] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.813724 16659 ts_tablet_manager.cc:1428] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260923 22:18:48.813787 16505 heartbeater.cc:503] Master 127.4.207.62:40489 was elected leader, sending a full tablet report...
Sep 23 22:18:48 dist-test-slave-7511 krb5kdc[15679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201928, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:18:48.872323 16663 raft_consensus.cc:482] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:18:48.872468 16663 raft_consensus.cc:504] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.872836 16663 leader_election.cc:288] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2ab47f5a184b1c97f64bc669f0a43a (127.4.207.3:36021), e924c06d7234431f9640a262b584bd1b (127.4.207.2:35475)
I20260923 22:18:48.876957 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.873282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:36587 (local address 127.4.207.3:36021)
0923 22:18:48.873331 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.873333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.873343 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:18:48.873430 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.873432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.873441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.873470 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:48.873698 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.874451 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.875955 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.876547 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.876591 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.876602 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.876840 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.876886 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:18:48.877377 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.873100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:36021 (local address 127.4.207.1:36587)
0923 22:18:48.873282 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.873292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.873340 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.873539 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:48.873540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:48.873542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:48.873631 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.873632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.874567 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.874568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.875850 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.875852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.876682 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.876683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.876766 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.876769 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:48.877253 (+ 484us) client_negotiation.cc:240] Negotiation successful
0923 22:18:48.877295 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260923 22:18:48.877758 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.873585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:41441 (local address 127.4.207.2:35475)
0923 22:18:48.873629 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.873632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.874451 (+ 819us) server_negotiation.cc:407] Connection header received
0923 22:18:48.874555 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.874557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.874570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.874598 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:18:48.874821 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.875551 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.876854 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.877393 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.877439 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.877447 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.877653 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.877690 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:18:48.877810 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08501a710c2b45a39ecf86d8430adf51" candidate_uuid: "c668153b01e24df9a40beb9dd38822ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" is_pre_election: true
I20260923 22:18:48.877990 16591 raft_consensus.cc:2454] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c668153b01e24df9a40beb9dd38822ed in term 0.
I20260923 22:18:48.877997 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.873532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:35475 (local address 127.4.207.1:41441)
0923 22:18:48.874380 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.874392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.874480 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.874662 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:18:48.874662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:18:48.874664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:18:48.874738 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.874740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.875688 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.875689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.876738 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:18:48.876753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.877508 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.877509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:18:48.877578 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.877581 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:18:48.877845 (+ 264us) client_negotiation.cc:240] Negotiation successful
0923 22:18:48.877892 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":100,"threads_started":1}
I20260923 22:18:48.878278 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08501a710c2b45a39ecf86d8430adf51" candidate_uuid: "c668153b01e24df9a40beb9dd38822ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e924c06d7234431f9640a262b584bd1b" is_pre_election: true
I20260923 22:18:48.878433 16459 raft_consensus.cc:2454] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c668153b01e24df9a40beb9dd38822ed in term 0.
I20260923 22:18:48.878472 16256 leader_election.cc:302] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f2ab47f5a184b1c97f64bc669f0a43a, c668153b01e24df9a40beb9dd38822ed; no voters:
I20260923 22:18:48.878599 16663 raft_consensus.cc:2781] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:18:48.878634 16663 raft_consensus.cc:482] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:18:48.878654 16663 raft_consensus.cc:3037] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:48.879662 16663 raft_consensus.cc:504] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.879787 16663 leader_election.cc:288] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [CANDIDATE]: Term 1 election: Requested vote from peers 8f2ab47f5a184b1c97f64bc669f0a43a (127.4.207.3:36021), e924c06d7234431f9640a262b584bd1b (127.4.207.2:35475)
I20260923 22:18:48.880147 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08501a710c2b45a39ecf86d8430adf51" candidate_uuid: "c668153b01e24df9a40beb9dd38822ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e924c06d7234431f9640a262b584bd1b"
I20260923 22:18:48.880223 16459 raft_consensus.cc:3037] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:48.880373 16591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08501a710c2b45a39ecf86d8430adf51" candidate_uuid: "c668153b01e24df9a40beb9dd38822ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a"
I20260923 22:18:48.880445 16591 raft_consensus.cc:3037] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:18:48.881028 16459 raft_consensus.cc:2454] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c668153b01e24df9a40beb9dd38822ed in term 1.
I20260923 22:18:48.881233 16257 leader_election.cc:302] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c668153b01e24df9a40beb9dd38822ed, e924c06d7234431f9640a262b584bd1b; no voters:
I20260923 22:18:48.881340 16591 raft_consensus.cc:2454] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c668153b01e24df9a40beb9dd38822ed in term 1.
I20260923 22:18:48.881371 16663 raft_consensus.cc:2781] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:18:48.881498 16663 raft_consensus.cc:686] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [term 1 LEADER]: Becoming Leader. State: Replica: c668153b01e24df9a40beb9dd38822ed, State: Running, Role: LEADER
I20260923 22:18:48.881579 16663 consensus_queue.cc:237] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } }
I20260923 22:18:48.882401 16155 catalog_manager.cc:5495] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed reported cstate change: term changed from 0 to 1, leader changed from <none> to c668153b01e24df9a40beb9dd38822ed (127.4.207.1). New cstate: current_term: 1 leader_uuid: "c668153b01e24df9a40beb9dd38822ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c668153b01e24df9a40beb9dd38822ed" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 } health_report { overall_health: UNKNOWN } } }
I20260923 22:18:48.895892 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.892770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.4.207.1:40475)
0923 22:18:48.892807 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:18:48.892809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:18:48.892815 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:18:48.892844 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:18:48.892845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:18:48.892877 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:18:48.892935 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:18:48.893105 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.893796 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.894860 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:18:48.895153 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:18:48.895216 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:18:48.895390 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:18:48.895522 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:18:48.895609 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 22:18:48.895772 (+ 163us) server_negotiation.cc:299] Negotiation successful
0923 22:18:48.895792 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:18:48.897760 16459 raft_consensus.cc:1261] T 08501a710c2b45a39ecf86d8430adf51 P e924c06d7234431f9640a262b584bd1b [term 1 FOLLOWER]: Refusing update from remote peer c668153b01e24df9a40beb9dd38822ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:18:48.898169 16663 consensus_queue.cc:1035] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e924c06d7234431f9640a262b584bd1b" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:18:48.898324 16591 raft_consensus.cc:1261] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 1 FOLLOWER]: Refusing update from remote peer c668153b01e24df9a40beb9dd38822ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:18:48.898779 16663 consensus_queue.cc:1035] T 08501a710c2b45a39ecf86d8430adf51 P c668153b01e24df9a40beb9dd38822ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f2ab47f5a184b1c97f64bc669f0a43a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 36021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:18:48.903373 16672 mvcc.cc:205] Tried to move back new op lower bound from 7332667100762333184 to 7332667100699475968. Current Snapshot: MvccSnapshot[applied={T|T < 7332667100762333184}]
I20260923 22:18:48.903928 16675 mvcc.cc:205] Tried to move back new op lower bound from 7332667100762333184 to 7332667100699475968. Current Snapshot: MvccSnapshot[applied={T|T < 7332667100762333184}]
I20260923 22:18:48.906356 16155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:18:48.906450 16155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:18:48.907111 16155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 487b48cca5784e19afda48f41cba430e: Sending DeleteTablet for 3 replicas of tablet 08501a710c2b45a39ecf86d8430adf51
I20260923 22:18:48.907509 16301 tablet_service.cc:1514] Processing DeleteTablet for tablet 08501a710c2b45a39ecf86d8430adf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:48 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:39152
I20260923 22:18:48.908196 16430 tablet_service.cc:1514] Processing DeleteTablet for tablet 08501a710c2b45a39ecf86d8430adf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:48 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:46086
I20260923 22:18:48.908612 16573 tablet_service.cc:1514] Processing DeleteTablet for tablet 08501a710c2b45a39ecf86d8430adf51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:18:48 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:39954
I20260923 22:18:48.909446 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 16236
I20260923 22:18:48.909862 16673 mvcc.cc:205] Tried to move back new op lower bound from 7332667100762333184 to 7332667100699475968. Current Snapshot: MvccSnapshot[applied={T|T < 7332667100762333184}]
W20260923 22:18:48.917896 16395 connection.cc:535] server connection from 127.4.207.1:41441 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 22:18:48.918335 16142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:40475 (error 108)
W20260923 22:18:48.918648 16140 connection.cc:535] server connection from 127.4.207.1:56409 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 22:18:48.919049 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 16373
I20260923 22:18:48.920578 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:18:48.920467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:40475 (local address 127.0.0.1:39166)
0923 22:18:48.920512 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.920543 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:40475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 22:18:48.920720 16142 catalog_manager.cc:4584] TS c668153b01e24df9a40beb9dd38822ed (127.4.207.1:40475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08501a710c2b45a39ecf86d8430adf51: Network error: Client connection negotiation failed: client connection to 127.4.207.1:40475: connect: Connection refused (error 111)
I20260923 22:18:48.920939 16682 tablet_replica.cc:330] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: stopping tablet replica
I20260923 22:18:48.921095 16682 raft_consensus.cc:2229] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:18:48.921196 16682 raft_consensus.cc:2258] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:18:48.922199 16682 ts_tablet_manager.cc:1905] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:18:48.924230 16682 ts_tablet_manager.cc:1918] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:18:48.924309 16682 log.cc:1196] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/08501a710c2b45a39ecf86d8430adf51
I20260923 22:18:48.924927 16682 ts_tablet_manager.cc:1939] T 08501a710c2b45a39ecf86d8430adf51 P 8f2ab47f5a184b1c97f64bc669f0a43a: Deleting consensus metadata
I20260923 22:18:48.925381 16141 catalog_manager.cc:4841] TS 8f2ab47f5a184b1c97f64bc669f0a43a (127.4.207.3:36021): tablet 08501a710c2b45a39ecf86d8430adf51 (table test-table [id=66c184ed71ca43169f01c91b4bf1e012]) successfully deleted
W20260923 22:18:48.925987 16655 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:18:48.925666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:35475 (local address 127.0.0.1:46098)
0923 22:18:48.925709 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 22:18:48.925720 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:18:48.925756 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:18:48.925952 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.2:35475: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 22:18:48.926446 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 16510
I20260923 22:18:48.933341 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 16123
I20260923 22:18:48.939172 4924 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 22:18:49.273378 4924 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 22:18:49.273481 4924 mini_ranger.cc:66] Stopping Ranger...
I20260923 22:18:49.618103 4924 mini_ranger.cc:68] Stopped Ranger
2026-09-23 22:18:49.618 UTC [15709] LOG: received smart shutdown request
2026-09-23 22:18:49.621 UTC [15709] LOG: background worker "logical replication launcher" (PID 15718) exited with exit code 1
2026-09-23 22:18:49.621 UTC [15713] LOG: shutting down
2026-09-23 22:18:49.794 UTC [15709] LOG: database system is shut down
2026-09-23T22:18:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45840 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:18:49 dist-test-slave-7511 krb5kdc[16693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:18:49 dist-test-slave-7511 krb5kdc[16693](info): set up 2 sockets
Sep 23 22:18:49 dist-test-slave-7511 krb5kdc[16693](info): commencing operation
krb5kdc: starting...
W20260923 22:18:50.939600 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:18:51 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:18:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:18:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.207.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.207.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/rangeradmin_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/rangeradmin_127.4.207.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.207.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.207.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/rangerlookup_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/rangerlookup_127.4.207.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.207.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.207.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/HTTP_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/HTTP_127.4.207.20@KRBTEST.COM.keytab.
I20260923 22:18:51.118573 4924 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMonjHj/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/postgres -l logfile start
2026-09-23 22:18:53.686 UTC [16723] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 22:18:53.686 UTC [16723] LOG: listening on IPv4 address "127.4.207.20", port 43745
2026-09-23 22:18:53.690 UTC [16723] LOG: listening on Unix socket "/tmp/.s.PGSQL.43745"
2026-09-23 22:18:53.725 UTC [16726] LOG: database system was shut down at 2026-09-23 22:18:52 UTC
2026-09-23 22:18:53.729 UTC [16723] LOG: database system is ready to accept connections
I20260923 22:18:54.667320 4924 mini_postgres.cc:96] Postgres bound to 43745
2026-09-23 22:18:54.673 UTC [16734] FATAL: database "slave" does not exist
127.4.207.20:43745 - accepting connections
I20260923 22:18:54.673769 4924 mini_ranger.cc:161] Starting Ranger...
I20260923 22:18:54.688647 4924 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 22:18:55.176185 4924 mini_ranger.cc:87] Created ranger Postgres database
I20260923 22:18:55.176276 4924 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-admin
2026-09-23 22:18:55,457 [I] DB FLAVOR :POSTGRES
2026-09-23 22:18:55,458 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 22:18:55,458 [I] Checking connection..
2026-09-23 22:18:55,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 22:18:55,721 [I] Checking connection passed.
2026-09-23 22:18:55,721 [I] --------- Verifying version history table ---------
2026-09-23 22:18:55,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:55,999 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 22:18:55,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:56,251 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 22:18:56,252 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 22:18:56,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 22:18:56.506 UTC [16798] WARNING: there is no transaction in progress
2026-09-23 22:18:56,512 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 22:18:56,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 22:18:56,774 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 22:18:56,775 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 22:18:56,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 22:18:57,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 22:18:57,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-7511.c.gcp-upstream.internal','N') ;"
2026-09-23 22:18:57,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 22:18:57,748 [I] Table x_portal_user does not exist in database ranger
2026-09-23 22:18:57,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 22:18:57,991 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 22:18:57,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 22:18:58,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 22:18:58,459 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 22:18:58,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 22:18:59.442 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.459 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.476 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.492 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.508 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.528 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.584 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.592 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.606 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.618 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.629 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.643 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.654 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.661 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.668 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:18:59.674 UTC [16906] WARNING: there is no transaction in progress
2026-09-23 22:19:00,314 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 22:19:00,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 22:19:00,541 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 22:19:00,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 22:19:00,770 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 22:19:00,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 22:19:01,021 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 22:19:01,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 22:19:01,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 22:19:01,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-7511.c.gcp-upstream.internal';"
2026-09-23 22:19:01,738 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 22:19:01,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 22:19:01,963 [I] DB_PATCHES have already been applied
I20260923 22:19:01.968488 4924 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMonjHj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 22:19:01.968566 4924 mini_ranger.cc:193] Using host: 127.4.207.20
I20260923 22:19:01.971694 4924 mini_ranger.cc:227] Ranger admin URL: http://127.4.207.20:58849
Sep 23, 2026 10:19:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 10:19:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 10:19:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 10:19:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 10:19:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 10:19:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 10:19:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 22:19:03.077860 4924 mini_ranger.cc:160] Time spent starting Ranger: real 8.404s user 0.001s sys 0.008s
Sep 23, 2026 10:19:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 10:19:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 10:19:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 10:19:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 10:19:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 22:19:23.600093 4924 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.207.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.207.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/rangerkms_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.207.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/rangerkms_127.4.207.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.207.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.207.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.207.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/HTTP_127.4.207.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.207.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/HTTP_127.4.207.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 22:19:23 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 22:19:23.685832 4924 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 22:19:23.695852 4924 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 22:19:24.653074 4924 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 22:19:24.653146 4924 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms
I20260923 22:19:24.653160 4924 mini_ranger_kms.cc:227] Using postgres at 127.4.207.20:43745
2026-09-23 22:19:24,812 [I] DB FLAVOR :POSTGRES
2026-09-23 22:19:24,812 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 22:19:24,812 [I] Checking connection
2026-09-23 22:19:24,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 22:19:25,046 [I] connection success
2026-09-23 22:19:25,047 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 22:19:25,047 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 22:19:25,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 22:19:25,280 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 22:19:25,281 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 22:19:25,281 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 22:19:25,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.207.20:43745/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 22:19:25,538 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 22:19:26.017239 4924 mini_ranger_kms.cc:310] Created kms service
I20260923 22:19:26.251262 4924 mini_ranger_kms.cc:326] Created kudu user
I20260923 22:19:26.296087 4924 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 22:19:26 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:26.701208 4924 mini_ranger_kms.cc:384] Added ranger policy
I20260923 22:19:26.701320 4924 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMonjHj/build/release/bin/postgresql.jar:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMonjHj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 22:19:26.701345 4924 mini_ranger_kms.cc:240] Using host: 127.4.207.20
I20260923 22:19:26.704716 4924 mini_ranger_kms.cc:276] Ranger KMS PID: 17086
I20260923 22:19:26.704799 4924 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.207.20:38483
Sep 23, 2026 10:19:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 10:19:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 10:19:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 10:19:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 10:19:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 10:19:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 22:19:28.849658 4924 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.164s user 0.007s sys 0.135s
I20260923 22:19:28.849766 4924 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 22:19:28.849823 4924 mini_ranger_kms.cc:401] 127.4.207.20:38483/kms/v1/keys
Sep 23 22:19:28 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.207.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 22:19:29 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.207.20@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
Sep 23 22:19:29 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.207.20@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
Sep 23 22:19:31 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:19:31.935039 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33359
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:33359
--ranger_config_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:38483/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:32.016021 17137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:32.016211 17137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:32.016232 17137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:32.017227 17137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:32.017257 17137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:32.017269 17137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:32.017282 17137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:32.017292 17137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:32.018590 17137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.207.20:38483/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:33359
--ranger_config_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:32.018750 17137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:32.018893 17137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:19:32.024024 17137 server_base.cc:1034] running on GCE node
W20260923 22:19:32.024097 17142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:32.023978 17143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:32.024106 17145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:32.024394 17137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:32.024701 17137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:32.025936 17137 hybrid_clock.cc:648] HybridClock initialized: now 1790201972025921 us; error 43 us; skew 500 ppm
Sep 23 22:19:32 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201972, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:32.034029 17137 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:19:32.034409 17137 webserver.cc:460] Webserver started at http://127.4.207.62:32853/ using document root <none> and password file <none>
I20260923 22:19:32.034593 17137 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:32.034626 17137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:32.034718 17137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:19:32 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201972, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:32.082470 17137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "1248631d353145cfa20f5c5739edaa82"
format_stamp: "Formatted at 2026-09-23 22:19:32 on dist-test-slave-7511"
server_key: "0f7eabca4d5ef17f32c51567b63a00eb"
server_key_iv: "2b7df1e1bbd6a082bc946d1d7752800d"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:32.082870 17137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "1248631d353145cfa20f5c5739edaa82"
format_stamp: "Formatted at 2026-09-23 22:19:32 on dist-test-slave-7511"
server_key: "0f7eabca4d5ef17f32c51567b63a00eb"
server_key_iv: "2b7df1e1bbd6a082bc946d1d7752800d"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:32.084472 17137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:19:32.096210 17152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:32.096410 17137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:32.096494 17137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "1248631d353145cfa20f5c5739edaa82"
format_stamp: "Formatted at 2026-09-23 22:19:32 on dist-test-slave-7511"
server_key: "0f7eabca4d5ef17f32c51567b63a00eb"
server_key_iv: "2b7df1e1bbd6a082bc946d1d7752800d"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:32.096603 17137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:32.108709 17137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:32.110772 17137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:32.110934 17137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:32.114665 17137 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:33359
I20260923 22:19:32.114701 17204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:33359 every 8 connection(s)
I20260923 22:19:32.115005 17137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:32.115701 17205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:32.118193 17205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Bootstrap starting.
I20260923 22:19:32.118798 17205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:32.119151 17205 log.cc:824] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:32.119921 17205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: No bootstrap required, opened a new log
I20260923 22:19:32.120770 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 17137
I20260923 22:19:32.120877 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:19:32.121049 4924 external_mini_cluster.cc:1428] Setting key 255481e06774db5518ef3f4d9c102ac1
I20260923 22:19:32.122393 17205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } }
I20260923 22:19:32.122539 17205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:32.122581 17205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1248631d353145cfa20f5c5739edaa82, State: Initialized, Role: FOLLOWER
I20260923 22:19:32.122749 17205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } }
I20260923 22:19:32.122841 17205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:32.122872 17205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:32.122910 17205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:32.123639 17205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } }
I20260923 22:19:32.123723 17205 leader_election.cc:302] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1248631d353145cfa20f5c5739edaa82; no voters:
I20260923 22:19:32.123908 17205 leader_election.cc:288] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:19:32.123937 17210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:32.124121 17210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 1 LEADER]: Becoming Leader. State: Replica: 1248631d353145cfa20f5c5739edaa82, State: Running, Role: LEADER
I20260923 22:19:32.124152 17205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:32.124286 17210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } }
I20260923 22:19:32.124640 17211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1248631d353145cfa20f5c5739edaa82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } } }
I20260923 22:19:32.124711 17211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:32.124948 17212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1248631d353145cfa20f5c5739edaa82. Latest consensus state: current_term: 1 leader_uuid: "1248631d353145cfa20f5c5739edaa82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } } }
I20260923 22:19:32.125270 17212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:32.125420 17217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:32.125669 17205 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 22:19:32.125954 17217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:32.131230 17217 catalog_manager.cc:1347] Generated new cluster ID: 3ce27e69eb3f4efabffca5013fe8f8f8
I20260923 22:19:32.131289 17217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:19:32 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:32.141139 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:32.121586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.4.207.62:33359)
0923 22:19:32.121744 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:32.121755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:32.121801 (+ 46us) server_negotiation.cc:407] Connection header received
0923 22:19:32.122412 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:32.122420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:32.122501 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:32.122668 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:32.123525 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:32.125055 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:32.127280 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:32.129122 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:32.129298 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:32.136027 (+ 6729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:32.136032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:32.136033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:32.136046 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:32.138584 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:32.139086 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:32.139090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:32.139091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:32.139203 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:32.139491 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:32.139492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:32.139492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:32.139898 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:32.140059 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:32.140682 (+ 623us) server_negotiation.cc:299] Negotiation successful
0923 22:19:32.140851 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260923 22:19:32.143602 17217 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:32.144415 17217 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:32.148710 17217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Generated new TSK 0
I20260923 22:19:32.148890 17217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:19:32.871706 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:38483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33359
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:32.978600 17250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:32.978828 17250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:32.978865 17250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:32.980062 17250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:32.980127 17250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:32.980170 17250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:19:32.981632 17250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.207.20:38483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:32.981878 17250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:32.982106 17250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:32.987012 17255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:32.987012 17256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:32.987579 17258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:32.987914 17250 server_base.cc:1034] running on GCE node
I20260923 22:19:32.988152 17250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:32.988472 17250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:32.989697 17250 hybrid_clock.cc:648] HybridClock initialized: now 1790201972989680 us; error 25 us; skew 500 ppm
Sep 23 22:19:32 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201972, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:32.992931 17250 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:19:32.993444 17250 webserver.cc:460] Webserver started at http://127.4.207.1:43823/ using document root <none> and password file <none>
I20260923 22:19:32.993731 17250 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:32.993793 17250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:32.993932 17250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:19:32 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201972, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:33.009161 17250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "135cf77dc4b6440c944b15d389e43792"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "c44b385287a385a68f7464f2989853d7"
server_key_iv: "f38e25e5acc37f62cd0e094cc889a4aa"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.011600 17250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "135cf77dc4b6440c944b15d389e43792"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "c44b385287a385a68f7464f2989853d7"
server_key_iv: "f38e25e5acc37f62cd0e094cc889a4aa"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.013705 17250 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:33.027915 17267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.028195 17250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:33.028312 17250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "135cf77dc4b6440c944b15d389e43792"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "c44b385287a385a68f7464f2989853d7"
server_key_iv: "f38e25e5acc37f62cd0e094cc889a4aa"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.028424 17250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:33.045878 17250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:33.048353 17250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:33.048549 17250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:33.048858 17250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:33.049296 17250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:33.049381 17250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.049479 17250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:33.049561 17250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.056595 17250 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:33385
I20260923 22:19:33.057122 17250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:33.058254 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 17250
I20260923 22:19:33.058357 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:19:33.058568 4924 external_mini_cluster.cc:1428] Setting key ee611278ad89af8ca55e4ed8b2b279fd
I20260923 22:19:33.057904 17380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:33385 every 8 connection(s)
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201972, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:33.084097 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.058524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33359 (local address 127.4.207.1:50111)
0923 22:19:33.059092 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.059138 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:33.059783 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:33.071250 (+ 11467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:33.071256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:33.071563 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:33.071904 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.071911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.073052 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.073053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.074436 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.074439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.075032 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.075033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.075194 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.075872 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:33.075883 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:33.077718 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:33.081918 (+ 4200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.081927 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.081930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.082314 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.082546 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.082546 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.082547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.082686 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.083323 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:33.083330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:33.083552 (+ 222us) client_negotiation.cc:769] Sending connection context
0923 22:19:33.083650 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 22:19:33.083815 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":86,"threads_started":1}
I20260923 22:19:33.084918 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.058553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:50111 (local address 127.4.207.62:33359)
0923 22:19:33.070930 (+ 12377us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:33.070933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:33.070948 (+ 15us) server_negotiation.cc:407] Connection header received
0923 22:19:33.070980 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:33.070981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:33.070997 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:33.072014 (+ 1017us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:33.072061 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.072922 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.074542 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.074906 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.075474 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.077825 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:33.077826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:33.077827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:33.077836 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:33.079800 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.082404 (+ 2604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.082405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.082406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.082473 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.082756 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.082757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.082757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.083235 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:33.084695 (+ 1460us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:33.084739 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 22:19:33.084828 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12335,"thread_start_us":87,"threads_started":1}
I20260923 22:19:33.085525 17381 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33359
I20260923 22:19:33.085628 17381 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:33.085822 17381 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:33.086453 17169 ts_manager.cc:194] Registered new tserver with Master: 135cf77dc4b6440c944b15d389e43792 (127.4.207.1:33385)
I20260923 22:19:33.087616 17169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:50111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:19:33.092427 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:38483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33359
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 22:19:33.224418 17387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:33.224833 17387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:33.224915 17387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:33.226744 17387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:33.226853 17387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:33.226935 17387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:19:33.229457 17387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.207.20:38483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:33.229943 17387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:33.230224 17387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:33.237087 17395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.237109 17394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.238212 17397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:33.238684 17387 server_base.cc:1034] running on GCE node
I20260923 22:19:33.238899 17387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:33.239255 17387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:33.240445 17387 hybrid_clock.cc:648] HybridClock initialized: now 1790201973240422 us; error 40 us; skew 500 ppm
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:33.243896 17387 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:19:33.244415 17387 webserver.cc:460] Webserver started at http://127.4.207.2:33993/ using document root <none> and password file <none>
I20260923 22:19:33.244692 17387 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:33.244745 17387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:33.244871 17387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:33.260625 17387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "ac8131f33b544648b072c166467ec5eb"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "8831484a77dd062a1f2dc97a2c16aeb8"
server_key_iv: "be09557957f756e6ba2c7b1e0faa0a8b"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.261188 17387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "ac8131f33b544648b072c166467ec5eb"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "8831484a77dd062a1f2dc97a2c16aeb8"
server_key_iv: "be09557957f756e6ba2c7b1e0faa0a8b"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.264556 17387 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260923 22:19:33.284518 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.284792 17387 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260923 22:19:33.284893 17387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "ac8131f33b544648b072c166467ec5eb"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "8831484a77dd062a1f2dc97a2c16aeb8"
server_key_iv: "be09557957f756e6ba2c7b1e0faa0a8b"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.285000 17387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:33.305208 17387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:33.307570 17387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:33.307741 17387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:33.308010 17387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:33.308420 17387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:33.308480 17387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.308557 17387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:33.308589 17387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.316318 17387 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:46657
I20260923 22:19:33.316702 17387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:19:33.317847 17517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:46657 every 8 connection(s)
I20260923 22:19:33.324046 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 17387
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:33.324156 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:19:33.324370 4924 external_mini_cluster.cc:1428] Setting key a21b62605df72c003507e350063c8492
I20260923 22:19:33.334158 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.317611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:45431 (local address 127.4.207.62:33359)
0923 22:19:33.317669 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:33.317671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:33.318341 (+ 670us) server_negotiation.cc:407] Connection header received
0923 22:19:33.319162 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:33.319163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:33.319178 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:33.319214 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:33.320030 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.320637 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.321842 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.322085 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.322174 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.324672 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:33.324674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:33.324674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:33.324682 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:33.326568 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.332742 (+ 6174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.332743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.332744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.332823 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.333165 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.333166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.333167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.333328 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:33.333388 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:33.333893 (+ 505us) server_negotiation.cc:299] Negotiation successful
0923 22:19:33.334003 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:19:33.334751 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.317618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33359 (local address 127.4.207.2:45431)
0923 22:19:33.318196 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.318239 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:33.318959 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:33.319427 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:33.319431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:33.319694 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:33.319898 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.319903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.320769 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.320771 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.321736 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.321739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.322206 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.322207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.322332 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.322915 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:33.322924 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:33.324531 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:33.326672 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.326679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.326681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.327019 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.332915 (+ 5896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.332916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.332916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.333079 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.333427 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:33.333434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:33.333658 (+ 224us) client_negotiation.cc:769] Sending connection context
0923 22:19:33.334280 (+ 622us) client_negotiation.cc:240] Negotiation successful
0923 22:19:33.334471 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":87,"threads_started":1}
I20260923 22:19:33.335259 17518 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33359
I20260923 22:19:33.335420 17518 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:33.335712 17518 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:33.336130 17169 ts_manager.cc:194] Registered new tserver with Master: ac8131f33b544648b072c166467ec5eb (127.4.207.2:46657)
I20260923 22:19:33.336665 17169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:45431
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:19:33.362205 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:38483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33359
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:33.479712 17524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:33.480119 17524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:33.480242 17524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:33.482184 17524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:33.482311 17524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:33.482405 17524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:19:33.484818 17524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.207.20:38483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:33.485317 17524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:33.485651 17524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:33.495510 17530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.495823 17532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.496131 17529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:33.497406 17524 server_base.cc:1034] running on GCE node
I20260923 22:19:33.497644 17524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:33.497942 17524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:33.499140 17524 hybrid_clock.cc:648] HybridClock initialized: now 1790201973499114 us; error 39 us; skew 500 ppm
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:33.503011 17524 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:19:33.503638 17524 webserver.cc:460] Webserver started at http://127.4.207.3:46149/ using document root <none> and password file <none>
I20260923 22:19:33.503993 17524 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:33.504122 17524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:33.504323 17524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:33.526166 17524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "fd2434799e124741bba93cf26b95b15a"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "34d8ce59b213b4b9d8efc92b549c7877"
server_key_iv: "b80d6ac07e82d93e62b59c8777adaac7"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.526696 17524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "fd2434799e124741bba93cf26b95b15a"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "34d8ce59b213b4b9d8efc92b549c7877"
server_key_iv: "b80d6ac07e82d93e62b59c8777adaac7"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.530149 17524 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260923 22:19:33.547436 17539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.547876 17524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 22:19:33.548038 17524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "fd2434799e124741bba93cf26b95b15a"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "34d8ce59b213b4b9d8efc92b549c7877"
server_key_iv: "b80d6ac07e82d93e62b59c8777adaac7"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.548156 17524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:33.580869 17524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:33.583505 17524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:33.583726 17524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:33.584039 17524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:33.584437 17524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:33.584493 17524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.584527 17524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:33.584596 17524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.593669 17524 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:39653
I20260923 22:19:33.593703 17652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:39653 every 8 connection(s)
I20260923 22:19:33.594393 17524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:33.605751 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 17524
I20260923 22:19:33.605865 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:19:33.606125 4924 external_mini_cluster.cc:1428] Setting key 1ef2e47398399e93f2c5e3017eb6525d
I20260923 22:19:33.606552 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.595342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:41247 (local address 127.4.207.62:33359)
0923 22:19:33.595397 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:33.595400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:33.595753 (+ 353us) server_negotiation.cc:407] Connection header received
0923 22:19:33.596416 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:33.596417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:33.596432 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:33.596530 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:33.597236 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.598103 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.599512 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.599846 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.599925 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.602878 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:33.602880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:33.602880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:33.602888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:33.604775 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.605325 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.605326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.605326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.605389 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:33.605708 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:33.605710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:33.605710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:33.605865 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:33.605950 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:33.606393 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 22:19:33.606487 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:19:33.607420 17655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:33.595305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33359 (local address 127.4.207.3:41247)
0923 22:19:33.595604 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:33.595645 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:33.596282 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:33.596604 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:33.596608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:33.596846 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:33.597112 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.597118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.598224 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.598226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.599366 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:33.599369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:33.599978 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:33.599980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:33.600130 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:33.600849 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:33.600860 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:33.602765 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:33.604890 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.604897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.604899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.605221 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.605471 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:33.605472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:33.605472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:33.605613 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:33.606012 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:33.606019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:33.606218 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 22:19:33.606881 (+ 663us) client_negotiation.cc:240] Negotiation successful
0923 22:19:33.607141 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":88,"threads_started":1}
I20260923 22:19:33.608060 17653 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33359
I20260923 22:19:33.608158 17653 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:33.608361 17653 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:33.608781 17169 ts_manager.cc:194] Registered new tserver with Master: fd2434799e124741bba93cf26b95b15a (127.4.207.3:39653)
I20260923 22:19:33.609254 17169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:41247
I20260923 22:19:33.618091 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:33.618762 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 17250
I20260923 22:19:33.625490 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 17387
I20260923 22:19:33.632956 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 17524
I20260923 22:19:33.638676 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 17137
I20260923 22:19:33.644479 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33359
--webserver_interface=127.4.207.62
--webserver_port=32853
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:33359
--ranger_config_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:38483/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.207.20:38483/kms with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:33.724284 17656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:33.724483 17656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:33.724503 17656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:33.725668 17656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:33.725709 17656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:33.725723 17656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:33.725734 17656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:33.725778 17656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:33.727236 17656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.207.20:38483/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:33359
--ranger_config_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=32853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:33.727512 17656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:33.727726 17656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:33.732355 17661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.732481 17664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:33.732695 17656 server_base.cc:1034] running on GCE node
W20260923 22:19:33.732410 17662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:33.732956 17656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:33.733187 17656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:33.734336 17656 hybrid_clock.cc:648] HybridClock initialized: now 1790201973734313 us; error 37 us; skew 500 ppm
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:33.736747 17656 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:19:33.737165 17656 webserver.cc:460] Webserver started at http://127.4.207.62:32853/ using document root <none> and password file <none>
I20260923 22:19:33.737371 17656 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:33.737418 17656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:33.798990 17656 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:33.799934 17672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.800133 17656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:33.800253 17656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "1248631d353145cfa20f5c5739edaa82"
format_stamp: "Formatted at 2026-09-23 22:19:32 on dist-test-slave-7511"
server_key: "0f7eabca4d5ef17f32c51567b63a00eb"
server_key_iv: "2b7df1e1bbd6a082bc946d1d7752800d"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.800657 17656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:33.812924 17656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:33.814723 17656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:33.814883 17656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:33.819229 17656 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:33359
I20260923 22:19:33.819267 17724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:33359 every 8 connection(s)
I20260923 22:19:33.819677 17656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:33.820204 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 17656
I20260923 22:19:33.820485 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:38483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:33385
--local_ip_for_outbound_sockets=127.4.207.1
--tserver_master_addrs=127.4.207.62:33359
--webserver_port=43823
--webserver_interface=127.4.207.1
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.207.20:38483/kms with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:33.821070 17725 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:19:33.821727 17725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Bootstrap starting.
I20260923 22:19:33.829469 17725 log.cc:824] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:33.831142 17725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:33.831415 17725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Bootstrap complete.
I20260923 22:19:33.833760 17725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } }
I20260923 22:19:33.833994 17725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1248631d353145cfa20f5c5739edaa82, State: Initialized, Role: FOLLOWER
I20260923 22:19:33.834167 17725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } }
I20260923 22:19:33.834225 17725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:33.834282 17725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:33.834329 17725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:19:33.835381 17725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } }
I20260923 22:19:33.835548 17725 leader_election.cc:302] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1248631d353145cfa20f5c5739edaa82; no voters:
I20260923 22:19:33.835784 17725 leader_election.cc:288] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 22:19:33.835816 17729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:19:33.835917 17729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [term 2 LEADER]: Becoming Leader. State: Replica: 1248631d353145cfa20f5c5739edaa82, State: Running, Role: LEADER
I20260923 22:19:33.836053 17725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:33.836009 17729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } }
I20260923 22:19:33.836351 17730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1248631d353145cfa20f5c5739edaa82. Latest consensus state: current_term: 2 leader_uuid: "1248631d353145cfa20f5c5739edaa82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } } }
I20260923 22:19:33.836452 17730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:33.836364 17731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1248631d353145cfa20f5c5739edaa82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1248631d353145cfa20f5c5739edaa82" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33359 } } }
I20260923 22:19:33.836614 17731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:33.837033 17733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:33.837502 17733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:33.837625 17733 catalog_manager.cc:1259] Loaded cluster ID: 3ce27e69eb3f4efabffca5013fe8f8f8
I20260923 22:19:33.837658 17733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:19:33.837858 17725 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 22:19:33.841221 17733 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:33.841750 17733 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Loaded TSK: 0
I20260923 22:19:33.842034 17733 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:19:33.927443 17727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:33.927744 17727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:33.927783 17727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:33.929117 17727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:33.929181 17727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:33.929224 17727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:19:33.931124 17727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.207.20:38483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:33385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=43823
--tserver_master_addrs=127.4.207.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:33.931391 17727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:33.931646 17727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:33.936405 17765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.936379 17763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:33.936576 17762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:33.937471 17727 server_base.cc:1034] running on GCE node
I20260923 22:19:33.937665 17727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:33.937892 17727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:33.941141 17727 hybrid_clock.cc:648] HybridClock initialized: now 1790201973941119 us; error 25 us; skew 500 ppm
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:33.944446 17727 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:19:33.944952 17727 webserver.cc:460] Webserver started at http://127.4.207.1:43823/ using document root <none> and password file <none>
I20260923 22:19:33.945228 17727 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:33.945302 17727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 22:19:33 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:33.961007 17727 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:33.961987 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:33.962203 17727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:33.962311 17727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "135cf77dc4b6440c944b15d389e43792"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "c44b385287a385a68f7464f2989853d7"
server_key_iv: "f38e25e5acc37f62cd0e094cc889a4aa"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:33.962697 17727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:34.009195 17727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:34.011339 17727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:34.011533 17727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:34.011801 17727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:34.012121 17727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:34.012176 17727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.012209 17727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:34.012235 17727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.018842 17727 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:33385
I20260923 22:19:34.018926 17886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:33385 every 8 connection(s)
I20260923 22:19:34.019265 17727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:34.026883 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 17727
I20260923 22:19:34.027172 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:38483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:46657
--local_ip_for_outbound_sockets=127.4.207.2
--tserver_master_addrs=127.4.207.62:33359
--webserver_port=33993
--webserver_interface=127.4.207.2
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.207.20:38483/kms with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:19:34 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201973, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:34.037233 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.020055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33359 (local address 127.4.207.1:39035)
0923 22:19:34.020318 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.020360 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:34.020886 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:34.021409 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:34.021412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:34.021754 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:34.022038 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.022044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.023731 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.023732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.024901 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.024905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.025950 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.025951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.026083 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.026641 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:34.026649 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:34.031753 (+ 5104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:34.034234 (+ 2481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.034240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.034243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.034580 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.035761 (+ 1181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.035762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.035762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.035899 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.036460 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:34.036466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:34.036691 (+ 225us) client_negotiation.cc:769] Sending connection context
0923 22:19:34.036817 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 22:19:34.036982 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":73,"threads_started":1}
I20260923 22:19:34.037811 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.020140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:39035 (local address 127.4.207.62:33359)
0923 22:19:34.020329 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.020338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.020451 (+ 113us) server_negotiation.cc:407] Connection header received
0923 22:19:34.021152 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.021159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.021234 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.021362 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.022191 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.023611 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.024994 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.025695 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.025832 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.031869 (+ 6037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.031874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.031874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.031887 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.034068 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.035540 (+ 1472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.035543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.035544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.035667 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.036000 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.036001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.036001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.036348 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.036431 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.037437 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.037567 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260923 22:19:34.038714 17887 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33359
I20260923 22:19:34.038817 17887 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.039023 17887 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
W20260923 22:19:34.040031 17887 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:33359 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 22:19:34.040122 17887 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.040195 17887 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:34.040604 17887 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.040680 17887 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
W20260923 22:19:34.041055 17887 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:19:34.139276 17891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:34.139526 17891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:34.139569 17891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:34.140784 17891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:34.140843 17891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:34.140887 17891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:19:34.142428 17891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.207.20:38483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:46657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=33993
--tserver_master_addrs=127.4.207.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:34.142665 17891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:34.142947 17891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:34.148900 17899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:34.149194 17897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:34.149557 17896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:34.150182 17891 server_base.cc:1034] running on GCE node
I20260923 22:19:34.150385 17891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:34.150663 17891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:34.151829 17891 hybrid_clock.cc:648] HybridClock initialized: now 1790201974151808 us; error 34 us; skew 500 ppm
Sep 23 22:19:34 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:34.154428 17891 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:19:34.154920 17891 webserver.cc:460] Webserver started at http://127.4.207.2:33993/ using document root <none> and password file <none>
I20260923 22:19:34.155196 17891 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:34.155297 17891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 22:19:34 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:34.176210 17891 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:34.177263 17907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.177435 17891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:34.177529 17891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "ac8131f33b544648b072c166467ec5eb"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "8831484a77dd062a1f2dc97a2c16aeb8"
server_key_iv: "be09557957f756e6ba2c7b1e0faa0a8b"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:34.177934 17891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:34.186537 17891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:34.189175 17891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:34.189344 17891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:34.189606 17891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:34.189954 17891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:34.190007 17891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.190073 17891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:34.190101 17891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.197454 17891 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:46657
I20260923 22:19:34.197552 18020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:46657 every 8 connection(s)
I20260923 22:19:34.197997 17891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:19:34.202082 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 17891
I20260923 22:19:34.202399 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.207.20:38483/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:39653
--local_ip_for_outbound_sockets=127.4.207.3
--tserver_master_addrs=127.4.207.62:33359
--webserver_port=46149
--webserver_interface=127.4.207.3
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.207.20:38483/kms with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 22:19:34 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:34.218336 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.198837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:57509 (local address 127.4.207.62:33359)
0923 22:19:34.198888 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.198891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.199129 (+ 238us) server_negotiation.cc:407] Connection header received
0923 22:19:34.199927 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.199928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.199945 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.200033 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.200881 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.201723 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.203367 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.203668 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.203731 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.214849 (+ 11118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.214851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.214852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.214861 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.216676 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.217209 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.217211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.217211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.217271 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.217548 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.217549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.217549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.217684 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.217725 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.218137 (+ 412us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.218233 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:19:34.219127 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.198817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33359 (local address 127.4.207.2:57509)
0923 22:19:34.198961 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.199013 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:34.199801 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:34.200128 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:34.200132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:34.200428 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:34.200766 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.200773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.201838 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.201839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.203266 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.203270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.203850 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.203851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.204014 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.204656 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:34.204666 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:34.214678 (+ 10012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:34.216772 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.216782 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.216785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.217111 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.217340 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.217340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.217340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.217466 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.217767 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:34.217773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:34.217962 (+ 189us) client_negotiation.cc:769] Sending connection context
0923 22:19:34.218670 (+ 708us) client_negotiation.cc:240] Negotiation successful
0923 22:19:34.218859 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260923 22:19:34.219861 18021 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33359
I20260923 22:19:34.219957 18021 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.220139 18021 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
W20260923 22:19:34.220896 18021 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:33359 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 22:19:34.220963 18021 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.221020 18021 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:34.221475 18021 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.221567 18021 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
W20260923 22:19:34.221977 18021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:19:34.305102 18024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:34.305352 18024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:34.305392 18024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:34.306567 18024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:34.306638 18024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:34.306681 18024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:19:34.308321 18024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.207.20:38483/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:39653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=46149
--tserver_master_addrs=127.4.207.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:34.308583 18024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:34.308802 18024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:34.314322 18030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:34.314535 18032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:34.314719 18024 server_base.cc:1034] running on GCE node
W20260923 22:19:34.315186 18029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:34.315516 18024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:34.315821 18024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:34.316975 18024 hybrid_clock.cc:648] HybridClock initialized: now 1790201974316965 us; error 30 us; skew 500 ppm
Sep 23 22:19:34 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:34.319708 18024 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:19:34.320214 18024 webserver.cc:460] Webserver started at http://127.4.207.3:46149/ using document root <none> and password file <none>
I20260923 22:19:34.320458 18024 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:34.320549 18024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 22:19:34 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:34.337968 18024 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:34.339886 18040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.340072 18024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:34.340188 18024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "fd2434799e124741bba93cf26b95b15a"
format_stamp: "Formatted at 2026-09-23 22:19:33 on dist-test-slave-7511"
server_key: "34d8ce59b213b4b9d8efc92b549c7877"
server_key_iv: "b80d6ac07e82d93e62b59c8777adaac7"
server_key_version: "kuduclusterkey@0"
I20260923 22:19:34.340567 18024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:34.357645 18024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:34.359884 18024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:34.360054 18024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:34.360306 18024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:34.360625 18024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:34.360679 18024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.360724 18024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:34.360750 18024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:34.367795 18024 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:39653
I20260923 22:19:34.367909 18154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:39653 every 8 connection(s)
I20260923 22:19:34.368254 18024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:19:34.369071 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 18024
Sep 23 22:19:34 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:34.380227 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.369167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:35991 (local address 127.4.207.62:33359)
0923 22:19:34.369245 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.369248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.369487 (+ 239us) server_negotiation.cc:407] Connection header received
0923 22:19:34.370347 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.370348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.370361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.370392 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.371151 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.371853 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.373255 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.373563 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.373652 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.376968 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.376969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.376969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.376976 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.378680 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.379162 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.379163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.379163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.379219 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.379519 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.379520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.379520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.379651 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.379696 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.380048 (+ 352us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.380129 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 22:19:34.380681 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.369183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33359 (local address 127.4.207.3:35991)
0923 22:19:34.369359 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:34.369401 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:34.370225 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:34.370587 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:34.370592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:34.370799 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:34.371053 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.371058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.371952 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.371953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.373162 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:34.373166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.373684 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.373686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:34.373822 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.374576 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:34.374585 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:34.376856 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:34.378773 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.378779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.378781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.379074 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.379286 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:34.379286 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:34.379286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:34.379406 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:34.379747 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:34.379753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:34.379900 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 22:19:34.380036 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 22:19:34.380182 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260923 22:19:34.381229 18155 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33359
I20260923 22:19:34.381374 18155 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.381559 18155 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
W20260923 22:19:34.382215 18155 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:33359 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 22:19:34.382314 18155 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.382386 18155 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:34.382709 18155 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:34.382797 18155 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
W20260923 22:19:34.383083 18155 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:19:34.384948 18159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:34.369618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.4.207.62:33359)
0923 22:19:34.369757 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:34.369759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:34.369909 (+ 150us) server_negotiation.cc:407] Connection header received
0923 22:19:34.369998 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:34.369999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:34.370014 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:34.370103 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:34.371535 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.372234 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.374096 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:34.374382 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:34.374444 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:34.381762 (+ 7318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:34.381763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:34.381763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:34.381770 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:34.383766 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.384182 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.384183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.384183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.384230 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:34.384460 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:34.384460 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:34.384461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:34.384554 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:34.384663 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:34.384795 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 22:19:34.384857 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
Sep 23 22:19:34 dist-test-slave-7511 krb5kdc[16693](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:35.041277 17887 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:35.041440 17887 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:35.041877 17689 ts_manager.cc:194] Registered new tserver with Master: 135cf77dc4b6440c944b15d389e43792 (127.4.207.1:33385)
I20260923 22:19:35.042891 17689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:39035
I20260923 22:19:35.222172 18021 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:35.222342 18021 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:35.222615 17689 ts_manager.cc:194] Registered new tserver with Master: ac8131f33b544648b072c166467ec5eb (127.4.207.2:46657)
I20260923 22:19:35.223161 17689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:57509
Sep 23 22:19:35 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
I20260923 22:19:35.383277 18155 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:35.383469 18155 heartbeater.cc:511] Master 127.4.207.62:33359 requested a full tablet report, sending...
I20260923 22:19:35.383817 17689 ts_manager.cc:194] Registered new tserver with Master: fd2434799e124741bba93cf26b95b15a (127.4.207.3:39653)
I20260923 22:19:35.384407 17689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:35991
I20260923 22:19:35.384660 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:35.394125 18177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.385675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.4.207.62:33359)
0923 22:19:35.385997 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:35.385999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:35.386008 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:35.386033 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:35.386034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:35.386047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:35.386124 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:35.386799 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.387695 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.388702 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.389071 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.389157 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.390645 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:35.390646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:35.390647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:35.390655 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:35.392667 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:35.393061 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:35.393062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:35.393063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:35.393130 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:35.393347 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:35.393348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:35.393348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:35.393515 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:35.393640 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:35.393785 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 22:19:35.393880 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":63,"threads_started":1}
I20260923 22:19:35.396437 17689 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:19:35.398792 17689 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:19:35.406965 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.401916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:33385 (local address 127.0.0.1:59414)
0923 22:19:35.402554 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:35.402562 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:35.402596 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:35.403306 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:35.403309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:35.403311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:35.403383 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.403385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.404606 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.404607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.405760 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.405763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.406696 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.406697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.406783 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.406801 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 22:19:35.406846 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 22:19:35.406894 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":92,"threads_started":1}
I20260923 22:19:35.408318 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.401886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:46657 (local address 127.0.0.1:60644)
0923 22:19:35.402240 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:35.402276 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:35.402328 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:35.403588 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:35.403590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:35.403592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:35.403648 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.403649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.405482 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.405483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.406663 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.406666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.408078 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.408079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.408143 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.408147 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:35.408200 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:19:35.408244 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":86,"threads_started":1}
I20260923 22:19:35.408728 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.402331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.4.207.2:46657)
0923 22:19:35.403379 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:35.403385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:35.403396 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:19:35.403429 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:35.403432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:35.403474 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:35.403769 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:35.403851 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.405370 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.406781 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.407969 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.408505 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.408579 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:35.408607 (+ 28us) server_negotiation.cc:299] Negotiation successful
0923 22:19:35.408651 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":59,"threads_started":1}
I20260923 22:19:35.409432 17952 tablet_service.cc:1467] Processing CreateTablet for tablet 02a19499661745e3b5d444a2788be782 (DEFAULT_TABLE table=test-table [id=8136fbc089f648dc8bf41632bef6caf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:35.409862 17952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a19499661745e3b5d444a2788be782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:35.411415 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.402126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:39653 (local address 127.0.0.1:53078)
0923 22:19:35.407326 (+ 5200us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:35.407334 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:35.407368 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:35.407662 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:35.407662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:35.407664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:35.407694 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.407695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.409184 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.409185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.410301 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.410304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.411231 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.411232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.411283 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.411286 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:19:35.411327 (+ 41us) client_negotiation.cc:240] Negotiation successful
0923 22:19:35.411359 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5190,"thread_start_us":58,"threads_started":1}
I20260923 22:19:35.411935 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.402170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59414 (local address 127.4.207.1:33385)
0923 22:19:35.403092 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:35.403097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:35.403109 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:19:35.403145 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:35.403147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:35.403163 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:35.403252 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:35.403492 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.404521 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.405855 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.406602 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.411740 (+ 5138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.411802 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:35.411826 (+ 24us) server_negotiation.cc:299] Negotiation successful
0923 22:19:35.411863 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":65,"threads_started":1}
I20260923 22:19:35.412235 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.402568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.4.207.3:39653)
0923 22:19:35.402694 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:35.402698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:35.407425 (+ 4727us) server_negotiation.cc:407] Connection header received
0923 22:19:35.407503 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:35.407505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:35.407521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:35.407589 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:35.407757 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.409096 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.410389 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.411090 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.411152 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.411199 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:35.412127 (+ 928us) server_negotiation.cc:299] Negotiation successful
0923 22:19:35.412167 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260923 22:19:35.412990 18089 tablet_service.cc:1467] Processing CreateTablet for tablet 02a19499661745e3b5d444a2788be782 (DEFAULT_TABLE table=test-table [id=8136fbc089f648dc8bf41632bef6caf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:35.413045 17821 tablet_service.cc:1467] Processing CreateTablet for tablet 02a19499661745e3b5d444a2788be782 (DEFAULT_TABLE table=test-table [id=8136fbc089f648dc8bf41632bef6caf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:35.413357 17821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a19499661745e3b5d444a2788be782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:35.413348 18089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02a19499661745e3b5d444a2788be782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:35.417444 18188 tablet_bootstrap.cc:492] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792: Bootstrap starting.
I20260923 22:19:35.418064 18188 tablet_bootstrap.cc:654] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:35.418349 18188 log.cc:824] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:35.418687 18189 tablet_bootstrap.cc:492] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb: Bootstrap starting.
I20260923 22:19:35.419219 18188 tablet_bootstrap.cc:492] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792: No bootstrap required, opened a new log
I20260923 22:19:35.419282 18188 ts_tablet_manager.cc:1397] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:35.420236 18189 tablet_bootstrap.cc:654] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:35.420642 18189 log.cc:824] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:35.421072 18190 tablet_bootstrap.cc:492] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: Bootstrap starting.
I20260923 22:19:35.421746 18190 tablet_bootstrap.cc:654] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:35.421800 18188 raft_consensus.cc:348] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.421970 18188 raft_consensus.cc:374] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:35.422011 18188 raft_consensus.cc:729] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135cf77dc4b6440c944b15d389e43792, State: Initialized, Role: FOLLOWER
I20260923 22:19:35.422143 18188 consensus_queue.cc:260] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.422400 18188 ts_tablet_manager.cc:1428] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:19:35.422430 18190 log.cc:824] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:35.422519 17887 heartbeater.cc:503] Master 127.4.207.62:33359 was elected leader, sending a full tablet report...
I20260923 22:19:35.423297 18190 tablet_bootstrap.cc:492] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: No bootstrap required, opened a new log
I20260923 22:19:35.423416 18190 ts_tablet_manager.cc:1397] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:35.423889 18189 tablet_bootstrap.cc:492] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb: No bootstrap required, opened a new log
I20260923 22:19:35.423954 18189 ts_tablet_manager.cc:1397] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260923 22:19:35.426223 18190 raft_consensus.cc:348] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.426461 18190 raft_consensus.cc:374] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:35.426409 18189 raft_consensus.cc:348] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.426574 18189 raft_consensus.cc:374] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:35.426613 18189 raft_consensus.cc:729] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8131f33b544648b072c166467ec5eb, State: Initialized, Role: FOLLOWER
I20260923 22:19:35.426760 18189 consensus_queue.cc:260] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.426559 18190 raft_consensus.cc:729] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2434799e124741bba93cf26b95b15a, State: Initialized, Role: FOLLOWER
I20260923 22:19:35.427021 18189 ts_tablet_manager.cc:1428] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:19:35.427131 18021 heartbeater.cc:503] Master 127.4.207.62:33359 was elected leader, sending a full tablet report...
I20260923 22:19:35.427109 18190 consensus_queue.cc:260] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.427505 18190 ts_tablet_manager.cc:1428] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 22:19:35.427670 18155 heartbeater.cc:503] Master 127.4.207.62:33359 was elected leader, sending a full tablet report...
W20260923 22:19:35.449476 18022 tablet.cc:2369] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 22:19:35 dist-test-slave-7511 krb5kdc[16693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201974, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for HTTP/127.4.207.20@KRBTEST.COM
W20260923 22:19:35.520805 17888 tablet.cc:2369] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:35.540314 18195 raft_consensus.cc:482] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:35.540472 18195 raft_consensus.cc:504] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.540879 18195 leader_election.cc:288] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 135cf77dc4b6440c944b15d389e43792 (127.4.207.1:33385), fd2434799e124741bba93cf26b95b15a (127.4.207.3:39653)
I20260923 22:19:35.545715 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.541428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:49553 (local address 127.4.207.1:33385)
0923 22:19:35.541475 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:35.541477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:35.541661 (+ 184us) server_negotiation.cc:407] Connection header received
0923 22:19:35.541808 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:35.541809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:35.541823 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:35.541859 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:35.542146 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.542924 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.544571 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.545208 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.545263 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.545275 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:35.545586 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 22:19:35.545634 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:19:35.546005 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.541089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:33385 (local address 127.4.207.2:49553)
0923 22:19:35.541588 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:35.541601 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:35.541713 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:35.541947 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:35.541948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:35.541950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:35.542041 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.542043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.543030 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.543031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.544453 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.544458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.545385 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.545386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.545483 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.545488 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:19:35.545852 (+ 364us) client_negotiation.cc:240] Negotiation successful
0923 22:19:35.545900 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":100,"threads_started":1}
I20260923 22:19:35.546332 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a19499661745e3b5d444a2788be782" candidate_uuid: "ac8131f33b544648b072c166467ec5eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "135cf77dc4b6440c944b15d389e43792" is_pre_election: true
I20260923 22:19:35.546504 17841 raft_consensus.cc:2454] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac8131f33b544648b072c166467ec5eb in term 0.
I20260923 22:19:35.546734 17909 leader_election.cc:302] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135cf77dc4b6440c944b15d389e43792, ac8131f33b544648b072c166467ec5eb; no voters:
I20260923 22:19:35.546881 18195 raft_consensus.cc:2781] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:35.546924 18195 raft_consensus.cc:482] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:35.546947 18195 raft_consensus.cc:3037] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:35.548314 18195 raft_consensus.cc:504] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.548504 18195 leader_election.cc:288] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [CANDIDATE]: Term 1 election: Requested vote from peers 135cf77dc4b6440c944b15d389e43792 (127.4.207.1:33385), fd2434799e124741bba93cf26b95b15a (127.4.207.3:39653)
I20260923 22:19:35.548717 17841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a19499661745e3b5d444a2788be782" candidate_uuid: "ac8131f33b544648b072c166467ec5eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "135cf77dc4b6440c944b15d389e43792"
I20260923 22:19:35.548797 17841 raft_consensus.cc:3037] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:35.549628 17841 raft_consensus.cc:2454] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac8131f33b544648b072c166467ec5eb in term 1.
I20260923 22:19:35.549840 17909 leader_election.cc:302] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 135cf77dc4b6440c944b15d389e43792, ac8131f33b544648b072c166467ec5eb; no voters:
I20260923 22:19:35.549973 18195 raft_consensus.cc:2781] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:35.550166 18195 raft_consensus.cc:686] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 1 LEADER]: Becoming Leader. State: Replica: ac8131f33b544648b072c166467ec5eb, State: Running, Role: LEADER
I20260923 22:19:35.550297 18195 consensus_queue.cc:237] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } }
I20260923 22:19:35.551177 17689 catalog_manager.cc:5495] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb reported cstate change: term changed from 0 to 1, leader changed from <none> to ac8131f33b544648b072c166467ec5eb (127.4.207.2). New cstate: current_term: 1 leader_uuid: "ac8131f33b544648b072c166467ec5eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8131f33b544648b072c166467ec5eb" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 46657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 } health_report { overall_health: UNKNOWN } } }
I20260923 22:19:35.559892 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.541426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:39867 (local address 127.4.207.3:39653)
0923 22:19:35.541492 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:35.541495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:35.545380 (+ 3885us) server_negotiation.cc:407] Connection header received
0923 22:19:35.546707 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:35.546708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:35.546723 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:35.546753 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:35.546941 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.547631 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.551523 (+ 3892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.552157 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.552234 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.552247 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:35.559758 (+ 7511us) server_negotiation.cc:299] Negotiation successful
0923 22:19:35.559818 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:19:35.560128 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.541307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:39653 (local address 127.4.207.2:39867)
0923 22:19:35.545305 (+ 3998us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:35.545318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:35.545409 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:35.546815 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:35.546816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:35.546818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:35.546871 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.546873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.547723 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.547724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.550769 (+ 3045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:35.550771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.559524 (+ 8753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.559525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:35.559637 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.559643 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 22:19:35.559991 (+ 348us) client_negotiation.cc:240] Negotiation successful
0923 22:19:35.560041 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3978,"thread_start_us":68,"threads_started":1}
I20260923 22:19:35.560497 18109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a19499661745e3b5d444a2788be782" candidate_uuid: "ac8131f33b544648b072c166467ec5eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd2434799e124741bba93cf26b95b15a" is_pre_election: true
I20260923 22:19:35.560681 18109 raft_consensus.cc:2454] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac8131f33b544648b072c166467ec5eb in term 0.
I20260923 22:19:35.561010 18108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02a19499661745e3b5d444a2788be782" candidate_uuid: "ac8131f33b544648b072c166467ec5eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd2434799e124741bba93cf26b95b15a"
I20260923 22:19:35.561110 18108 raft_consensus.cc:3037] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:35.562120 18108 raft_consensus.cc:2454] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac8131f33b544648b072c166467ec5eb in term 1.
I20260923 22:19:35.569954 18186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.566495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.4.207.2:46657)
0923 22:19:35.566534 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:35.566536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:35.566556 (+ 20us) server_negotiation.cc:407] Connection header received
0923 22:19:35.566589 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:35.566590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:35.566613 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:35.566677 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:35.566860 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.567631 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.568776 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:35.569055 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:35.569112 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:35.569312 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:35.569454 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:35.569576 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:35.569851 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 22:19:35.569876 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:19:35.572422 18108 raft_consensus.cc:1261] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 1 FOLLOWER]: Refusing update from remote peer ac8131f33b544648b072c166467ec5eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:35.572780 18195 consensus_queue.cc:1035] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd2434799e124741bba93cf26b95b15a" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:35.574805 17841 raft_consensus.cc:1261] T 02a19499661745e3b5d444a2788be782 P 135cf77dc4b6440c944b15d389e43792 [term 1 FOLLOWER]: Refusing update from remote peer ac8131f33b544648b072c166467ec5eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:35.575088 18195 consensus_queue.cc:1035] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "135cf77dc4b6440c944b15d389e43792" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 33385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:35.579219 18205 mvcc.cc:205] Tried to move back new op lower bound from 7332667291941969920 to 7332667291854606336. Current Snapshot: MvccSnapshot[applied={T|T < 7332667291941969920}]
I20260923 22:19:35.580160 18207 mvcc.cc:205] Tried to move back new op lower bound from 7332667291941969920 to 7332667291854606336. Current Snapshot: MvccSnapshot[applied={T|T < 7332667291941969920}]
I20260923 22:19:35.582679 17689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:35.582778 17689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:35.583426 17689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1248631d353145cfa20f5c5739edaa82: Sending DeleteTablet for 3 replicas of tablet 02a19499661745e3b5d444a2788be782
I20260923 22:19:35.583894 17952 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a19499661745e3b5d444a2788be782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:35 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:60644
I20260923 22:19:35.584106 18089 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a19499661745e3b5d444a2788be782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:35 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:53078
I20260923 22:19:35.584460 17821 tablet_service.cc:1514] Processing DeleteTablet for tablet 02a19499661745e3b5d444a2788be782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:35 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:59414
I20260923 22:19:35.585052 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 17727
I20260923 22:19:35.587891 18204 mvcc.cc:205] Tried to move back new op lower bound from 7332667291941969920 to 7332667291854606336. Current Snapshot: MvccSnapshot[applied={T|T < 7332667291941969920 or (T in {7332667291941969920})}]
I20260923 22:19:35.595484 18214 tablet_replica.cc:330] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: stopping tablet replica
I20260923 22:19:35.595623 18214 raft_consensus.cc:2229] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:35.595750 18214 raft_consensus.cc:2258] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:35.595928 18213 tablet_replica.cc:330] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb: stopping tablet replica
I20260923 22:19:35.596040 18213 raft_consensus.cc:2229] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:19:35.596174 18213 raft_consensus.cc:2258] T 02a19499661745e3b5d444a2788be782 P ac8131f33b544648b072c166467ec5eb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260923 22:19:35.596678 17674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:33385 (error 108)
I20260923 22:19:35.597314 18214 ts_tablet_manager.cc:1905] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:35.599421 18214 ts_tablet_manager.cc:1918] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:35.599522 18214 log.cc:1196] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/02a19499661745e3b5d444a2788be782
I20260923 22:19:35.599830 18214 ts_tablet_manager.cc:1939] T 02a19499661745e3b5d444a2788be782 P fd2434799e124741bba93cf26b95b15a: Deleting consensus metadata
I20260923 22:19:35.600286 17673 catalog_manager.cc:4841] TS fd2434799e124741bba93cf26b95b15a (127.4.207.3:39653): tablet 02a19499661745e3b5d444a2788be782 (table test-table [id=8136fbc089f648dc8bf41632bef6caf2]) successfully deleted
I20260923 22:19:35.600533 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 17891
I20260923 22:19:35.612684 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 18024
I20260923 22:19:35.612709 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.612529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:33385 (local address 127.0.0.1:59426)
0923 22:19:35.612595 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:35.612652 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:33385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 22:19:35.612841 17674 catalog_manager.cc:4584] TS 135cf77dc4b6440c944b15d389e43792 (127.4.207.1:33385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02a19499661745e3b5d444a2788be782: Network error: Client connection negotiation failed: client connection to 127.4.207.1:33385: connect: Connection refused (error 111)
I20260923 22:19:35.616847 18182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:35.616740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:46657 (local address 127.0.0.1:60660)
0923 22:19:35.616790 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:35.616814 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.2:46657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 22:19:35.621054 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 17656
I20260923 22:19:35.628225 4924 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 22:19:35.972769 4924 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 22:19:35.972865 4924 mini_ranger.cc:66] Stopping Ranger...
I20260923 22:19:36.308681 4924 mini_ranger.cc:68] Stopped Ranger
2026-09-23 22:19:36.308 UTC [16723] LOG: received smart shutdown request
2026-09-23 22:19:36.312 UTC [16723] LOG: background worker "logical replication launcher" (PID 16732) exited with exit code 1
2026-09-23 22:19:36.313 UTC [16727] LOG: shutting down
2026-09-23 22:19:36.478 UTC [16723] LOG: database system is shut down
2026-09-23T22:19:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46675 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:19:36 dist-test-slave-7511 krb5kdc[18225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:19:36 dist-test-slave-7511 krb5kdc[18225](info): set up 2 sockets
Sep 23 22:19:36 dist-test-slave-7511 krb5kdc[18225](info): commencing operation
krb5kdc: starting...
W20260923 22:19:37.599112 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:19:37 dist-test-slave-7511 krb5kdc[18225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:19:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:19:37.710352 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35333
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35333,127.4.207.61:39107,127.4.207.60:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:37.787591 18238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:37.787806 18238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:37.787828 18238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:37.788997 18238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:37.789036 18238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:37.789050 18238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:37.789063 18238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:37.789075 18238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:37.790482 18238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35333,127.4.207.61:39107,127.4.207.60:36313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:37.790678 18238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:37.790840 18238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:37.795532 18244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:37.795737 18238 server_base.cc:1034] running on GCE node
W20260923 22:19:37.795535 18243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:37.795535 18246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:37.796139 18238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:37.796446 18238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:37.797763 18238 hybrid_clock.cc:648] HybridClock initialized: now 1790201977797736 us; error 43 us; skew 500 ppm
Sep 23 22:19:37 dist-test-slave-7511 krb5kdc[18225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:37.803493 18238 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:19:37.803881 18238 webserver.cc:460] Webserver started at http://127.4.207.62:45219/ using document root <none> and password file <none>
I20260923 22:19:37.804072 18238 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:37.804126 18238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:37.804215 18238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:37.805253 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "6f305014c5494005b38b29c320aac55c"
format_stamp: "Formatted at 2026-09-23 22:19:37 on dist-test-slave-7511"
server_key: "101152e14ac209ed51bb3b0e823583eb"
server_key_iv: "a8cfb508d731c7c43a9218db057fc3b1"
server_key_version: "encryptionkey@0"
I20260923 22:19:37.805563 18238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "6f305014c5494005b38b29c320aac55c"
format_stamp: "Formatted at 2026-09-23 22:19:37 on dist-test-slave-7511"
server_key: "101152e14ac209ed51bb3b0e823583eb"
server_key_iv: "a8cfb508d731c7c43a9218db057fc3b1"
server_key_version: "encryptionkey@0"
I20260923 22:19:37.806777 18238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:19:37.807596 18253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:37.807776 18238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:37.807861 18238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "6f305014c5494005b38b29c320aac55c"
format_stamp: "Formatted at 2026-09-23 22:19:37 on dist-test-slave-7511"
server_key: "101152e14ac209ed51bb3b0e823583eb"
server_key_iv: "a8cfb508d731c7c43a9218db057fc3b1"
server_key_version: "encryptionkey@0"
I20260923 22:19:37.807961 18238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:37.820217 18238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:37.822424 18238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:37.822594 18238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:37.826556 18238 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:35333
I20260923 22:19:37.826586 18305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:35333 every 8 connection(s)
I20260923 22:19:37.826928 18238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:37.827836 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:37.829391 18306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:37.834817 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 18238
I20260923 22:19:37.834933 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
Sep 23 22:19:37 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:37.835150 4924 external_mini_cluster.cc:1428] Setting key 3a3b78cb60e823c77b911124a81fa9c1
I20260923 22:19:37.838665 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.830127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.4.207.62:35333)
0923 22:19:37.830335 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:37.830339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:37.830381 (+ 42us) server_negotiation.cc:407] Connection header received
0923 22:19:37.830892 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:37.830893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:37.830909 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:37.830992 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:37.831540 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.832325 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.833152 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.833469 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.833559 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:37.835705 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:37.835711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:37.835711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:37.835717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:37.837225 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:37.837637 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:37.837639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:37.837640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:37.837705 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:37.837924 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:37.837925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:37.837925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:37.838108 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:37.838163 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:37.838361 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 22:19:37.838525 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260923 22:19:37.838768 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.830103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35333 (local address 127.0.0.1:47162)
0923 22:19:37.830281 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.830315 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:37.830756 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:37.831009 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:37.831011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:37.831272 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:37.831417 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:37.831421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.832412 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.832413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:37.833062 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:37.833064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.833548 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.833549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:37.833604 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:37.834053 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:37.834061 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:37.835628 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:37.837309 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:37.837315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:37.837317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:37.837564 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:37.837764 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:37.837765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:37.837766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:37.837868 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:37.838173 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:37.838179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:37.838249 (+ 70us) client_negotiation.cc:769] Sending connection context
0923 22:19:37.838348 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 22:19:37.838609 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":120,"thread_start_us":86,"threads_started":1}
I20260923 22:19:37.839470 18306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:37.839815 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.839690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40636)
0923 22:19:37.839737 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.839786 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 22:19:37.839951 18256 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
I20260923 22:19:37.840380 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.840298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40638)
0923 22:19:37.840324 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.840352 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260923 22:19:37.840488 18306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:39107: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.207.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab.
Entry for principal kudu/127.4.207.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab.
I20260923 22:19:37.858583 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.61:39107
--webserver_interface=127.4.207.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab
--principal=kudu/127.4.207.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35333,127.4.207.61:39107,127.4.207.60:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:37.889711 18306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } attempt: 1
I20260923 22:19:37.890323 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.890150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40654)
0923 22:19:37.890233 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.890284 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 22:19:37.890698 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.890584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40666)
0923 22:19:37.890635 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.890663 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 22:19:37.890789 18306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:39107: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
W20260923 22:19:37.940500 18313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:37.940709 18313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:37.940735 18313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:37.941916 18313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:37.941959 18313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:37.941973 18313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:37.941985 18313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:37.942004 18313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:37.943521 18313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35333,127.4.207.61:39107,127.4.207.60:36313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab
--principal=kudu/127.4.207.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.61:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.207.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:37.943771 18313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:37.944000 18313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:37.948818 18318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:37.948887 18313 server_base.cc:1034] running on GCE node
W20260923 22:19:37.948797 18319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:37.948784 18321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:37.949280 18313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:37.949592 18313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:37.950738 18313 hybrid_clock.cc:648] HybridClock initialized: now 1790201977950725 us; error 30 us; skew 500 ppm
Sep 23 22:19:37 dist-test-slave-7511 krb5kdc[18225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:37.953193 18313 init.cc:377] Logged in from keytab as kudu/127.4.207.61@KRBTEST.COM (short username kudu)
I20260923 22:19:37.953637 18313 webserver.cc:460] Webserver started at http://127.4.207.61:34805/ using document root <none> and password file <none>
I20260923 22:19:37.953811 18313 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:37.953850 18313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:37.953933 18313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:37.954890 18313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/data/instance:
uuid: "c3785b4a495f400d84e025ac75465b66"
format_stamp: "Formatted at 2026-09-23 22:19:37 on dist-test-slave-7511"
server_key: "48a9b475cd7fef61369708117c46019f"
server_key_iv: "f9c9481d54582bbe2a60dd65fbe566c0"
server_key_version: "encryptionkey@0"
I20260923 22:19:37.955191 18313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/wal/instance:
uuid: "c3785b4a495f400d84e025ac75465b66"
format_stamp: "Formatted at 2026-09-23 22:19:37 on dist-test-slave-7511"
server_key: "48a9b475cd7fef61369708117c46019f"
server_key_iv: "f9c9481d54582bbe2a60dd65fbe566c0"
server_key_version: "encryptionkey@0"
I20260923 22:19:37.956496 18313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:19:37.957188 18328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:37.957370 18313 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:19:37.957443 18313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/wal
uuid: "c3785b4a495f400d84e025ac75465b66"
format_stamp: "Formatted at 2026-09-23 22:19:37 on dist-test-slave-7511"
server_key: "48a9b475cd7fef61369708117c46019f"
server_key_iv: "f9c9481d54582bbe2a60dd65fbe566c0"
server_key_version: "encryptionkey@0"
I20260923 22:19:37.957537 18313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:37.958916 18306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } attempt: 2
I20260923 22:19:37.959283 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.959168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40674)
0923 22:19:37.959222 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.959257 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:19:37.959652 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.959535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40680)
0923 22:19:37.959587 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.959624 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 22:19:37.959759 18306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:39107: Network error: Client connection negotiation failed: client connection to 127.4.207.61:39107: connect: Connection refused (error 111)
I20260923 22:19:37.969727 18313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:37.971694 18313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:37.971845 18313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:37.975780 18313 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.61:39107
I20260923 22:19:37.975811 18380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.61:39107 every 8 connection(s)
I20260923 22:19:37.976172 18313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
I20260923 22:19:37.976974 18381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:37.978499 18381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:37.983309 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 18313
I20260923 22:19:37.983423 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-1/wal/instance
I20260923 22:19:37.983682 4924 external_mini_cluster.cc:1428] Setting key 62839e5fe755c54b1cbd223b566c2bb5
Sep 23 22:19:37 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:37.987692 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.979238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.4.207.62:35333)
0923 22:19:37.979307 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:37.979308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:37.979521 (+ 213us) server_negotiation.cc:407] Connection header received
0923 22:19:37.979948 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:37.979950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:37.979963 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:37.980046 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:37.980627 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.981111 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.982038 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.982236 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.982288 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:37.984547 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:37.984548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:37.984548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:37.984555 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:37.985984 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:37.986510 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:37.986512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:37.986512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:37.986570 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:37.986858 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:37.986859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:37.986859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:37.986965 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:37.987018 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:37.987407 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 22:19:37.987554 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:19:37.987872 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.979204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35333 (local address 127.0.0.1:47168)
0923 22:19:37.979384 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.979424 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:37.979870 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:37.980125 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:37.980128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:37.980366 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:37.980551 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:37.980555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.981200 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.981201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:37.981955 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:37.981956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.982307 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.982307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:37.982398 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:37.982891 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:37.982898 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:37.984431 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:37.986089 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:37.986097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:37.986098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:37.986407 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:37.986646 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:37.986647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:37.986648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:37.986792 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:37.987034 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:37.987041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:37.987204 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 22:19:37.987357 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 22:19:37.987583 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260923 22:19:37.988535 18381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.207.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.60@KRBTEST.COM" created.
Sep 23 22:19:37 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:19:37.998957 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.988863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40694)
0923 22:19:37.988919 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:37.988932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:37.988980 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:37.989196 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:37.989197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:37.989395 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:37.989470 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:37.989472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.990328 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.990329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:37.991196 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:37.991199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.991697 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.991698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:37.991781 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:37.992220 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:37.992226 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:37.995743 (+ 3517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:37.997869 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:37.997870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:37.997871 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:37.998023 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:37.998249 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:37.998250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:37.998250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:37.998310 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:37.998713 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:37.998715 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:37.998763 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 22:19:37.998829 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:19:37.998897 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 22:19:37.999083 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.988899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.4.207.61:39107)
0923 22:19:37.989063 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:37.989066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:37.989080 (+ 14us) server_negotiation.cc:407] Connection header received
0923 22:19:37.989098 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:37.989100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:37.989111 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:37.989171 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:37.989548 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.990233 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.991284 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:37.991622 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:37.991674 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:37.995825 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:37.995833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:37.995833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:37.995840 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:37.997775 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:37.998101 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:37.998104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:37.998105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:37.998165 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:37.998376 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:37.998377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:37.998378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:37.998592 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:37.998683 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:37.998890 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 22:19:37.998972 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260923 22:19:37.999696 18381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:38.000078 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:37.999924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46626)
0923 22:19:37.999977 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.000046 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 22:19:38.000238 18329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
I20260923 22:19:38.000736 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.000605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46642)
0923 22:19:38.000683 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.000712 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 22:19:38.000847 18381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:36313: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab.
Entry for principal kudu/127.4.207.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab.
I20260923 22:19:38.010022 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.60:36313
--webserver_interface=127.4.207.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab
--principal=kudu/127.4.207.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35333,127.4.207.61:39107,127.4.207.60:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:38.050096 18381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } attempt: 1
I20260923 22:19:38.050668 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.050505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46658)
0923 22:19:38.050573 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.050628 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 22:19:38.050899 18306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } attempt: 3
I20260923 22:19:38.051129 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.051019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46672)
0923 22:19:38.051065 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.051091 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 22:19:38.051224 18381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:36313: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:19:38.058012 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.051306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.4.207.61:39107)
0923 22:19:38.051342 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.051344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.051370 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:19:38.051423 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.051424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.051459 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.051534 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.051851 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.052347 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.053027 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.053237 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.053312 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.055353 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.055354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.055354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.055361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.056806 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.057172 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.057173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.057173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.057232 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.057447 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.057448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.057449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.057586 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.057673 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.057884 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.057952 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:19:38.058051 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.051221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40704)
0923 22:19:38.051316 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.051328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.051367 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.051543 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.051544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.051704 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.051768 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.051770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.052430 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.052431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.052933 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.052936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.053309 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.053310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.053392 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.053897 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.053904 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.055258 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.056916 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.056917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.056918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.057096 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.057318 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.057319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.057320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.057385 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.057694 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.057695 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.057753 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.057847 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.057951 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 22:19:38.058521 18306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:38.062263 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.062103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46676)
0923 22:19:38.062168 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.062195 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 22:19:38.063303 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.063208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46682)
0923 22:19:38.063236 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.063260 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260923 22:19:38.063395 18306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:36313: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
I20260923 22:19:38.094549 18306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } attempt: 1
W20260923 22:19:38.094815 18388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260923 22:19:38.095045 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.094879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46698)
0923 22:19:38.094956 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.095006 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 22:19:38.095119 18388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:38.095160 18388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 22:19:38.095520 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.095377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46712)
0923 22:19:38.095459 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.095495 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 22:19:38.095654 18306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:36313: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
W20260923 22:19:38.096511 18388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:38.096583 18388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:38.096611 18388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:38.096658 18388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:38.096690 18388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:38.098183 18388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35333,127.4.207.61:39107,127.4.207.60:36313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab
--principal=kudu/127.4.207.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.60:36313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.207.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:38.098459 18388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:38.098701 18388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:38.103811 18393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.103864 18396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:38.103822 18394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:38.103823 18388 server_base.cc:1034] running on GCE node
I20260923 22:19:38.104308 18388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:38.104647 18388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:38.105803 18388 hybrid_clock.cc:648] HybridClock initialized: now 1790201978105798 us; error 49 us; skew 500 ppm
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:38.108605 18388 init.cc:377] Logged in from keytab as kudu/127.4.207.60@KRBTEST.COM (short username kudu)
I20260923 22:19:38.109040 18388 webserver.cc:460] Webserver started at http://127.4.207.60:44181/ using document root <none> and password file <none>
I20260923 22:19:38.109261 18388 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:38.109328 18388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:38.109486 18388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:38.110574 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/data/instance:
uuid: "1d38913043114f348798ac3b2a431724"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-7511"
server_key: "7259c362e227fab5831be17430685e68"
server_key_iv: "983816a2036498abfdbd8120c4fbfb97"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.110953 18388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/wal/instance:
uuid: "1d38913043114f348798ac3b2a431724"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-7511"
server_key: "7259c362e227fab5831be17430685e68"
server_key_iv: "983816a2036498abfdbd8120c4fbfb97"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.112336 18388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:19:38.113098 18403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:38.113284 18388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:38.113385 18388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/wal
uuid: "1d38913043114f348798ac3b2a431724"
format_stamp: "Formatted at 2026-09-23 22:19:38 on dist-test-slave-7511"
server_key: "7259c362e227fab5831be17430685e68"
server_key_iv: "983816a2036498abfdbd8120c4fbfb97"
server_key_version: "encryptionkey@0"
I20260923 22:19:38.113452 18388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:38.119366 18381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } attempt: 2
I20260923 22:19:38.119864 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.119730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46728)
0923 22:19:38.119789 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.119829 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 22:19:38.120234 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.120141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46736)
0923 22:19:38.120185 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.120212 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:19:38.120357 18381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:36313: Network error: Client connection negotiation failed: client connection to 127.4.207.60:36313: connect: Connection refused (error 111)
I20260923 22:19:38.134135 18388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:38.136086 18388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:38.136238 18388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:38.140323 18388 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.60:36313
I20260923 22:19:38.140407 18455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.60:36313 every 8 connection(s)
I20260923 22:19:38.140769 18388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
I20260923 22:19:38.141777 18456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:38.143748 18456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:38.145105 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 18388
I20260923 22:19:38.145215 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790201827590652-4924-0/minicluster-data/master-2/wal/instance
I20260923 22:19:38.145378 4924 external_mini_cluster.cc:1428] Setting key 5873e948c80dd09fa931cb5e1a427442
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:19:38.153620 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.144432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.4.207.62:35333)
0923 22:19:38.144515 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.144518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.144689 (+ 171us) server_negotiation.cc:407] Connection header received
0923 22:19:38.145120 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.145122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.145134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.145214 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.146074 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.146585 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.147637 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.147868 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.147917 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.150169 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.150171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.150171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.150179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.152152 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.152635 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.152636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.152636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.152684 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.152947 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.152948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.152948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.153053 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.153102 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.153430 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.153508 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 22:19:38.154011 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.144402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35333 (local address 127.0.0.1:47174)
0923 22:19:38.144575 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.144609 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.145030 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.145296 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.145299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.145684 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.145973 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.145981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.146675 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.146676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.147557 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.147559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.147942 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.147943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.148086 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.148697 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.148705 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.150083 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.152227 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.152234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.152237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.152560 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.152749 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.152750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.152750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.152882 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.153117 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.153122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.153274 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.153391 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.153745 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260923 22:19:38.154599 18456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:38.155014 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.145751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46740 (local address 127.4.207.60:36313)
0923 22:19:38.145892 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.145897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.145944 (+ 47us) server_negotiation.cc:407] Connection header received
0923 22:19:38.146014 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.146016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.146062 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.146141 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.146748 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.147623 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.148600 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.148930 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.148984 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.151641 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.151647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.151648 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.151653 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.153342 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.153737 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.153739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.153740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.153798 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.154018 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.154019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.154019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.154198 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.154340 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.154431 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.154520 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:38.161458 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.154850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:39107 (local address 127.0.0.1:40716)
0923 22:19:38.154897 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.154907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.154943 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.155200 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.155201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.155328 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.155378 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.155380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.156120 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.156121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.156717 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.156719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.157130 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.157131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.157198 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.157509 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.157514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.158790 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.160442 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.160443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.160443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.160610 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.160816 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.160816 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.160816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.160856 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.161147 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.161148 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.161191 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.161280 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.161369 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 22:19:38.161473 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.154926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.4.207.61:39107)
0923 22:19:38.154965 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.154967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.154977 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:19:38.155001 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.155002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.155018 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.155088 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.155495 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.156047 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.156787 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.157065 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.157113 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.158873 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.158874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.158875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.158882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.160316 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.160684 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.160686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.160687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.160748 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.160925 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.160926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.160927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.161083 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.161148 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.161318 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.161390 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:19:38.161762 18456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:19:38.163214 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.155760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.4.207.62:35333)
0923 22:19:38.155797 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.155799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.155808 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:38.155829 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.155830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.155839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.155893 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.156418 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.156885 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.157735 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.157902 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.157952 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.160116 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.160118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.160118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.160125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.162124 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.162490 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.162491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.162491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.162562 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.162721 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.162721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.162722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.162843 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.163015 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.163048 (+ 33us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.163129 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201978, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:19:38.168685 18458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.162009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46746)
0923 22:19:38.162059 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.162068 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.162101 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.162306 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.162307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.162446 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.162484 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.162486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.163316 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.163317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.164117 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.164120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.164440 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.164441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.164497 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.164902 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.164907 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.166013 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.167654 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.167654 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.167655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.167816 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.168008 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.168009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.168009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.168066 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.168357 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.168358 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.168398 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.168475 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.168608 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 22:19:38.168784 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.162091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.4.207.60:36313)
0923 22:19:38.162217 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.162219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.162224 (+ 5us) server_negotiation.cc:407] Connection header received
0923 22:19:38.162238 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.162239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.162245 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.162284 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.162556 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.163226 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.164194 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.164372 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.164416 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.166105 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.166106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.166106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.166113 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.167571 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.167894 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.167895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.167895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.167944 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.168133 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.168133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.168134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.168269 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.168340 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.168492 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.168689 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":109}
I20260923 22:19:38.170327 18456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724: Bootstrap starting.
I20260923 22:19:38.170800 18306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } attempt: 2
I20260923 22:19:38.171272 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.163804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40722 (local address 127.4.207.61:39107)
0923 22:19:38.163836 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.163838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.163846 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:19:38.163865 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.163865 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.163873 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.163919 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.164486 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.164892 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.165677 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.165938 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.166018 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.168371 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.168372 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.168373 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.168380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.170068 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.170442 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.170443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.170444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.170499 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.170696 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.170697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.170697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.170832 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.171031 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.171075 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.171168 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:19:38.171489 18456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:38.171819 18456 log.cc:824] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:38.172606 18456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724: No bootstrap required, opened a new log
I20260923 22:19:38.173023 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:19:38.174808 18456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
I20260923 22:19:38.174960 18456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:38.174997 18456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d38913043114f348798ac3b2a431724, State: Initialized, Role: FOLLOWER
I20260923 22:19:38.175135 18456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:19:38.175420 18456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:38.177384 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.178619 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.171055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46752)
0923 22:19:38.171139 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.171148 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.171195 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.171429 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.171431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.171599 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.171660 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.171662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.172685 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.172686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.173410 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.173413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.173875 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.173875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.173931 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.174359 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.174363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.175687 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.177520 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.177521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.177522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.177751 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.177971 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.177972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.177972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.178024 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.178324 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.178325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.178366 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.178427 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.178508 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 22:19:38.178953 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.171258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.4.207.60:36313)
0923 22:19:38.171298 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.171299 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.171308 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:38.171326 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.171327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.171336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.171816 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.171846 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.172580 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.173506 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.173779 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.174207 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.175773 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.175774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.175774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.175780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.177442 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.177835 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.177836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.177836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.177903 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.178091 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.178092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.178092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.178243 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.178300 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.178791 (+ 491us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.178859 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:19:38.180413 18306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c: Bootstrap starting.
I20260923 22:19:38.181221 18306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:38.177469 18388 master_runner.cc:186] Error getting master registration for 127.4.207.62:35333: OK, instance_id { permanent_uuid: "6f305014c5494005b38b29c320aac55c" instance_seqno: 1790201977825441 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:19:38.181541 18306 log.cc:824] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:38.181583 18388 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:19:38.177472 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:19:38.182341 18481 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:38.182395 18481 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:38.183056 18306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c: No bootstrap required, opened a new log
I20260923 22:19:38.183553 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.174749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.4.207.62:35333)
0923 22:19:38.174821 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.174823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.174830 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:38.174846 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.174846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.174854 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.174897 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.175800 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.176488 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.177809 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.178080 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.178138 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.180013 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.180014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.180014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.180020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.181903 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.182312 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.182314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.182314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.182365 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.182683 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.182684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.182685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.182803 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.182887 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.183362 (+ 475us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.183461 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 22:19:38.184630 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.174624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.4.207.60:36313)
0923 22:19:38.177177 (+ 2553us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.177179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.177186 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:38.177203 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.177204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.177214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.177272 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.178159 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.178909 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.179801 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.180140 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.180204 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.181573 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.181574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.181575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.181581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.183312 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.183707 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.183708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.183708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.183761 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.183940 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.183941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.183941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.184048 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.184143 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.184347 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.184516 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2524,"thread_start_us":63,"threads_started":1}
I20260923 22:19:38.185104 18306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
I20260923 22:19:38.185247 18306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:38.185295 18306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f305014c5494005b38b29c320aac55c, State: Initialized, Role: FOLLOWER
I20260923 22:19:38.185413 18306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
I20260923 22:19:38.185657 18484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.185747 18484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:38.185746 18306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: configured and running, proceeding with master startup.
W20260923 22:19:38.187510 18495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:38.187619 18495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:38.189134 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.174784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.4.207.61:39107)
0923 22:19:38.174817 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.174819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.174826 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:38.174845 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.174845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.174854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.174897 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.175580 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.176493 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.183006 (+ 6513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.183289 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.183329 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.184833 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.184834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.184835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.184842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.187702 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.188078 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.188079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.188079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.188138 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.188355 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.188357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.188357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.188478 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.188613 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.188974 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.189046 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 22:19:38.188927 18238 master_runner.cc:186] Error getting master registration for 127.4.207.61:39107: OK, instance_id { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" instance_seqno: 1790201977974646 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:19:38.190694 18238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:19:38.211514 18381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } attempt: 3
Sep 23 22:19:38 dist-test-slave-7511 krb5kdc[18225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201977, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:19:38.219650 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.211874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:36313 (local address 127.0.0.1:46770)
0923 22:19:38.211932 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:38.211956 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:38.212011 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:38.212158 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:38.212159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:38.212343 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:38.212390 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.212391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.213148 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.213149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.213644 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:38.213646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.214090 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.214091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:38.214151 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.214717 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:38.214723 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:38.216162 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:38.218213 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.218214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.218215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.218413 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.218966 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:38.218967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:38.218967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:38.219034 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:38.219334 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:38.219335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:38.219382 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 22:19:38.219475 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 22:19:38.219581 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 22:19:38.220104 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:38.211913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.4.207.60:36313)
0923 22:19:38.211958 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:38.211960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:38.211992 (+ 32us) server_negotiation.cc:407] Connection header received
0923 22:19:38.212078 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:38.212079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:38.212090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:38.212153 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:38.212503 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.213030 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.213744 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:38.213998 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:38.214092 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:38.216319 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:38.216320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:38.216321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:38.216328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:38.218102 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.218813 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.218814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.218814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.218881 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:38.219113 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:38.219114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:38.219115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:38.219253 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:38.219860 (+ 607us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:38.219923 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 22:19:38.220012 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:19:38.221347 18381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66: Bootstrap starting.
I20260923 22:19:38.221987 18381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:38.222292 18381 log.cc:824] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:38.223117 18381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66: No bootstrap required, opened a new log
I20260923 22:19:38.224692 18381 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
I20260923 22:19:38.224812 18381 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:38.224851 18381 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3785b4a495f400d84e025ac75465b66, State: Initialized, Role: FOLLOWER
I20260923 22:19:38.225005 18381 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
I20260923 22:19:38.225291 18498 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.225351 18381 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:38.225417 18498 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:19:38.227702 18509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:38.227797 18509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:38.229645 18313 master_runner.cc:263] No leader master found from master c3785b4a495f400d84e025ac75465b66
I20260923 22:19:38.229697 18313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:19:38.309933 18484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:38.310073 18484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
I20260923 22:19:38.310309 18484 leader_election.cc:288] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3785b4a495f400d84e025ac75465b66 (127.4.207.61:39107), 1d38913043114f348798ac3b2a431724 (127.4.207.60:36313)
I20260923 22:19:38.310770 18430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f305014c5494005b38b29c320aac55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d38913043114f348798ac3b2a431724" is_pre_election: true
I20260923 22:19:38.310937 18430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f305014c5494005b38b29c320aac55c in term 0.
I20260923 22:19:38.311183 18254 leader_election.cc:302] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d38913043114f348798ac3b2a431724, 6f305014c5494005b38b29c320aac55c; no voters:
I20260923 22:19:38.311545 18484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:38.311628 18484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:38.311666 18484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:38.312624 18484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
I20260923 22:19:38.312762 18484 leader_election.cc:288] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [CANDIDATE]: Term 1 election: Requested vote from peers c3785b4a495f400d84e025ac75465b66 (127.4.207.61:39107), 1d38913043114f348798ac3b2a431724 (127.4.207.60:36313)
I20260923 22:19:38.313071 18430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f305014c5494005b38b29c320aac55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d38913043114f348798ac3b2a431724"
I20260923 22:19:38.313151 18430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:38.314054 18430 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f305014c5494005b38b29c320aac55c in term 1.
I20260923 22:19:38.314211 18254 leader_election.cc:302] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d38913043114f348798ac3b2a431724, 6f305014c5494005b38b29c320aac55c; no voters:
I20260923 22:19:38.314594 18355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f305014c5494005b38b29c320aac55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3785b4a495f400d84e025ac75465b66" is_pre_election: true
I20260923 22:19:38.314745 18355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f305014c5494005b38b29c320aac55c in term 0.
I20260923 22:19:38.314852 18354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f305014c5494005b38b29c320aac55c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3785b4a495f400d84e025ac75465b66"
I20260923 22:19:38.314910 18354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:38.315759 18354 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f305014c5494005b38b29c320aac55c in term 1.
I20260923 22:19:38.316111 18484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:38.316257 18484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [term 1 LEADER]: Becoming Leader. State: Replica: 6f305014c5494005b38b29c320aac55c, State: Running, Role: LEADER
I20260923 22:19:38.316346 18484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } }
I20260923 22:19:38.316718 18484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f305014c5494005b38b29c320aac55c. Latest consensus state: current_term: 1 leader_uuid: "6f305014c5494005b38b29c320aac55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.316810 18484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:38.317690 18513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:38.317862 18513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:38.319376 18430 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [term 1 FOLLOWER]: Refusing update from remote peer 6f305014c5494005b38b29c320aac55c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:38.319717 18484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:38.322024 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f305014c5494005b38b29c320aac55c. Latest consensus state: current_term: 1 leader_uuid: "6f305014c5494005b38b29c320aac55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.322124 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:38.322547 18354 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [term 1 FOLLOWER]: Refusing update from remote peer 6f305014c5494005b38b29c320aac55c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:38.322772 18484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:19:38.323402 18259 tablet.cc:2369] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:38.324465 18511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f305014c5494005b38b29c320aac55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.324450 18498 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f305014c5494005b38b29c320aac55c. Latest consensus state: current_term: 1 leader_uuid: "6f305014c5494005b38b29c320aac55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.324540 18511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:38.324550 18498 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:38.324865 18470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f305014c5494005b38b29c320aac55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.324941 18470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:38.325109 18514 mvcc.cc:205] Tried to move back new op lower bound from 7332667303193968640 to 7332667303184166912. Current Snapshot: MvccSnapshot[applied={T|T < 7332667303193968640}]
I20260923 22:19:38.325552 18515 mvcc.cc:205] Tried to move back new op lower bound from 7332667303193968640 to 7332667303184166912. Current Snapshot: MvccSnapshot[applied={T|T < 7332667303193968640}]
I20260923 22:19:38.325969 18513 catalog_manager.cc:1347] Generated new cluster ID: 48c499ece2074c378980b95de4bfa497
I20260923 22:19:38.326015 18513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:19:38.336076 18484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f305014c5494005b38b29c320aac55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.336167 18484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:38.336524 18498 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f305014c5494005b38b29c320aac55c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f305014c5494005b38b29c320aac55c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35333 } } peers { permanent_uuid: "c3785b4a495f400d84e025ac75465b66" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 39107 } } peers { permanent_uuid: "1d38913043114f348798ac3b2a431724" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 36313 } } }
I20260923 22:19:38.336602 18498 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:19:38.337195 18517 mvcc.cc:205] Tried to move back new op lower bound from 7332667303193968640 to 7332667303184166912. Current Snapshot: MvccSnapshot[applied={T|T < 7332667303193968640}]
I20260923 22:19:38.358273 18513 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:38.359063 18513 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:38.363489 18513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f305014c5494005b38b29c320aac55c: Generated new TSK 0
I20260923 22:19:38.363657 18513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:19:39.182812 18481 catalog_manager.cc:1259] Loaded cluster ID: 48c499ece2074c378980b95de4bfa497
I20260923 22:19:39.182886 18481 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724: loading cluster ID for follower catalog manager: success
I20260923 22:19:39.183714 18481 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724: acquiring CA information for follower catalog manager: success
I20260923 22:19:39.184011 18481 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d38913043114f348798ac3b2a431724: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:19:39.228238 18509 catalog_manager.cc:1259] Loaded cluster ID: 48c499ece2074c378980b95de4bfa497
I20260923 22:19:39.228312 18509 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66: loading cluster ID for follower catalog manager: success
I20260923 22:19:39.229107 18509 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66: acquiring CA information for follower catalog manager: success
I20260923 22:19:39.229408 18509 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3785b4a495f400d84e025ac75465b66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:19:39.424863 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 18238
I20260923 22:19:39.430213 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 18313
I20260923 22:19:39.435362 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 18388
2026-09-23T22:19:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2853 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:19:39 dist-test-slave-7511 krb5kdc[18528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:19:39 dist-test-slave-7511 krb5kdc[18528](info): set up 2 sockets
Sep 23 22:19:39 dist-test-slave-7511 krb5kdc[18528](info): commencing operation
krb5kdc: starting...
W20260923 22:19:40.447656 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:19:40 dist-test-slave-7511 krb5kdc[18528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:19:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:19:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:19:40.566847 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33731
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:37991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:33731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:40.646081 18541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:40.646340 18541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:40.646375 18541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:40.647588 18541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:40.647650 18541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:40.647673 18541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:40.647701 18541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:40.647732 18541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:40.649205 18541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:37991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:33731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:40.649478 18541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:40.649706 18541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:40.654651 18546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:40.654743 18547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:40.654781 18549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:40.654672 18541 server_base.cc:1034] running on GCE node
I20260923 22:19:40.655057 18541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:40.655398 18541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:40.656589 18541 hybrid_clock.cc:648] HybridClock initialized: now 1790201980656578 us; error 28 us; skew 500 ppm
Sep 23 22:19:40 dist-test-slave-7511 krb5kdc[18528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201980, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:40.659080 18541 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:19:40.659636 18541 webserver.cc:460] Webserver started at http://127.4.207.62:41873/ using document root <none> and password file <none>
I20260923 22:19:40.659865 18541 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:40.659910 18541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:40.660065 18541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:40.661131 18541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "cc6a27f8b4be45509d1d7337cc49ad79"
format_stamp: "Formatted at 2026-09-23 22:19:40 on dist-test-slave-7511"
server_key: "ed2687d58e55ef7f6b5554dcb116d3f8"
server_key_iv: "3fd26a5a67b10b7480feeb54f51ce981"
server_key_version: "encryptionkey@0"
I20260923 22:19:40.661511 18541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "cc6a27f8b4be45509d1d7337cc49ad79"
format_stamp: "Formatted at 2026-09-23 22:19:40 on dist-test-slave-7511"
server_key: "ed2687d58e55ef7f6b5554dcb116d3f8"
server_key_iv: "3fd26a5a67b10b7480feeb54f51ce981"
server_key_version: "encryptionkey@0"
I20260923 22:19:40.662882 18541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:19:40.663722 18556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:40.663910 18541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:40.664002 18541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "cc6a27f8b4be45509d1d7337cc49ad79"
format_stamp: "Formatted at 2026-09-23 22:19:40 on dist-test-slave-7511"
server_key: "ed2687d58e55ef7f6b5554dcb116d3f8"
server_key_iv: "3fd26a5a67b10b7480feeb54f51ce981"
server_key_version: "encryptionkey@0"
I20260923 22:19:40.664093 18541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:40.670388 18541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:40.672238 18541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:40.672379 18541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:40.676433 18541 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:33731
I20260923 22:19:40.676487 18608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:33731 every 8 connection(s)
I20260923 22:19:40.676795 18541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:40.677597 18609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:40.679807 18609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79: Bootstrap starting.
I20260923 22:19:40.680321 18609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:40.680553 18609 log.cc:824] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:40.681164 18609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79: No bootstrap required, opened a new log
I20260923 22:19:40.681432 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 18541
I20260923 22:19:40.681529 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:19:40.681730 4924 external_mini_cluster.cc:1428] Setting key c70cadffa47fc555417f7ef69b3cf9d2
I20260923 22:19:40.683302 18609 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6a27f8b4be45509d1d7337cc49ad79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33731 } }
I20260923 22:19:40.683521 18609 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:40.683614 18609 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6a27f8b4be45509d1d7337cc49ad79, State: Initialized, Role: FOLLOWER
I20260923 22:19:40.683779 18609 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6a27f8b4be45509d1d7337cc49ad79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33731 } }
I20260923 22:19:40.683873 18609 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:40.683911 18609 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:40.684003 18609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:40.684645 18609 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6a27f8b4be45509d1d7337cc49ad79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33731 } }
I20260923 22:19:40.684764 18609 leader_election.cc:302] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc6a27f8b4be45509d1d7337cc49ad79; no voters:
I20260923 22:19:40.684927 18609 leader_election.cc:288] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:19:40.684976 18614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:40.685122 18614 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [term 1 LEADER]: Becoming Leader. State: Replica: cc6a27f8b4be45509d1d7337cc49ad79, State: Running, Role: LEADER
I20260923 22:19:40.685171 18609 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:40.685249 18614 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6a27f8b4be45509d1d7337cc49ad79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33731 } }
I20260923 22:19:40.685667 18615 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc6a27f8b4be45509d1d7337cc49ad79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6a27f8b4be45509d1d7337cc49ad79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33731 } } }
I20260923 22:19:40.685735 18615 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:40.685926 18616 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc6a27f8b4be45509d1d7337cc49ad79. Latest consensus state: current_term: 1 leader_uuid: "cc6a27f8b4be45509d1d7337cc49ad79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc6a27f8b4be45509d1d7337cc49ad79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33731 } } }
I20260923 22:19:40.686040 18616 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79 [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:40.686375 18621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:40.686889 18621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:40.688539 18621 catalog_manager.cc:1347] Generated new cluster ID: 95e7dff49bb145e3a772066c04f61b38
I20260923 22:19:40.688596 18621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:19:40 dist-test-slave-7511 krb5kdc[18528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:40.693521 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:40.682230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56978 (local address 127.4.207.62:33731)
0923 22:19:40.682432 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:40.682439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:40.682466 (+ 27us) server_negotiation.cc:407] Connection header received
0923 22:19:40.683284 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:40.683291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:40.683336 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:40.683492 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:40.684300 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:40.685813 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:40.686922 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:40.687568 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:40.687688 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:40.690162 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:40.690166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:40.690166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:40.690178 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:40.691931 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:40.692331 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:40.692336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:40.692336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:40.692434 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:40.692658 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:40.692659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:40.692659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:40.692925 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:40.693115 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:40.693188 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 22:19:40.693332 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260923 22:19:40.694444 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:19:40.709483 18621 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:40.710191 18621 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:40.721665 18621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc6a27f8b4be45509d1d7337cc49ad79: Generated new TSK 0
I20260923 22:19:40.721856 18621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:19:40.748149 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 18541
2026-09-23T22:19:40Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1311 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:19:40 dist-test-slave-7511 krb5kdc[18636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:19:40 dist-test-slave-7511 krb5kdc[18636](info): set up 2 sockets
Sep 23 22:19:40 dist-test-slave-7511 krb5kdc[18636](info): commencing operation
krb5kdc: starting...
W20260923 22:19:41.771488 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:19:41 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:19:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:19:41.890636 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42257
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:41.968603 18649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:41.968816 18649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:41.968850 18649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:41.969949 18649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:41.970013 18649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:41.970038 18649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:41.970069 18649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:41.970090 18649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:41.971657 18649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:41.971910 18649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:41.972128 18649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:41.976729 18654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:41.976723 18655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:41.976804 18649 server_base.cc:1034] running on GCE node
W20260923 22:19:41.976737 18657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:41.977187 18649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:41.977505 18649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:41.978688 18649 hybrid_clock.cc:648] HybridClock initialized: now 1790201981978683 us; error 38 us; skew 500 ppm
Sep 23 22:19:41 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201981, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:41.981295 18649 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:19:41.981786 18649 webserver.cc:460] Webserver started at http://127.4.207.62:38217/ using document root <none> and password file <none>
I20260923 22:19:41.982041 18649 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:41.982115 18649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:41.982264 18649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:41.983289 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "88f233cbd5f44b8fb81bcebe3b5b388c"
format_stamp: "Formatted at 2026-09-23 22:19:41 on dist-test-slave-7511"
server_key: "80c6ae96b3dbb1deaa3f8f934bbe514c"
server_key_iv: "cfda1236979e65647546dee16c720c1e"
server_key_version: "encryptionkey@0"
I20260923 22:19:41.983750 18649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "88f233cbd5f44b8fb81bcebe3b5b388c"
format_stamp: "Formatted at 2026-09-23 22:19:41 on dist-test-slave-7511"
server_key: "80c6ae96b3dbb1deaa3f8f934bbe514c"
server_key_iv: "cfda1236979e65647546dee16c720c1e"
server_key_version: "encryptionkey@0"
I20260923 22:19:41.985029 18649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:41.985728 18664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:41.985912 18649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:41.986008 18649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "88f233cbd5f44b8fb81bcebe3b5b388c"
format_stamp: "Formatted at 2026-09-23 22:19:41 on dist-test-slave-7511"
server_key: "80c6ae96b3dbb1deaa3f8f934bbe514c"
server_key_iv: "cfda1236979e65647546dee16c720c1e"
server_key_version: "encryptionkey@0"
I20260923 22:19:41.986097 18649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:42.011296 18649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:42.013449 18649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:42.013631 18649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:42.017724 18649 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:42257
I20260923 22:19:42.017760 18716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:42257 every 8 connection(s)
I20260923 22:19:42.018122 18649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:42.019047 18717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:42.021281 18717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Bootstrap starting.
I20260923 22:19:42.021870 18717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:42.022181 18717 log.cc:824] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:42.022851 18717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: No bootstrap required, opened a new log
I20260923 22:19:42.024356 18717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:42.024482 18717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:42.024503 18717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f233cbd5f44b8fb81bcebe3b5b388c, State: Initialized, Role: FOLLOWER
I20260923 22:19:42.024681 18717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:42.024760 18717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:42.024806 18717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:42.024852 18717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:42.025175 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 18649
I20260923 22:19:42.025247 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:19:42.025427 4924 external_mini_cluster.cc:1428] Setting key aaec84bc99f19bf48015a5b961947b66
I20260923 22:19:42.025717 18717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:42.025835 18717 leader_election.cc:302] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f233cbd5f44b8fb81bcebe3b5b388c; no voters:
I20260923 22:19:42.025995 18717 leader_election.cc:288] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:19:42.026167 18723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:42.026269 18723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 1 LEADER]: Becoming Leader. State: Replica: 88f233cbd5f44b8fb81bcebe3b5b388c, State: Running, Role: LEADER
I20260923 22:19:42.026268 18717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:42.026422 18723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:42.027603 18724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } } }
I20260923 22:19:42.027627 18721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f233cbd5f44b8fb81bcebe3b5b388c. Latest consensus state: current_term: 1 leader_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } } }
I20260923 22:19:42.027688 18724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:42.027706 18721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: This master's current role is: LEADER
W20260923 22:19:42.028301 18737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:19:42.028353 18737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:19:42.028404 18738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:42.028537 18738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:42.030085 18738 catalog_manager.cc:1347] Generated new cluster ID: 3db2b12ec87749c2a377e033d971a7f5
I20260923 22:19:42.030138 18738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:42.034843 18738 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:19:42.035499 18738 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:42.036458 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.025893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.4.207.62:42257)
0923 22:19:42.026063 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.026073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.026109 (+ 36us) server_negotiation.cc:407] Connection header received
0923 22:19:42.026262 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.026269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.026334 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.027226 (+ 892us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:42.027343 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.028716 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.029683 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.030231 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.030356 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.032723 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:42.032727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:42.032727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:42.032738 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:42.034933 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.035286 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.035289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.035290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.035381 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.035620 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.035621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.035621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.035842 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:42.036034 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.036104 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.036251 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260923 22:19:42.043593 18738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Generated new TSK 0
I20260923 22:19:42.043763 18738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:19:42.070766 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42257
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:42.149331 18743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:42.149544 18743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:42.149566 18743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:42.150741 18743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:42.150789 18743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:42.150816 18743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:19:42.152331 18743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:42.152604 18743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:42.152873 18743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:42.157944 18748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.157940 18751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.157940 18749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:42.158198 18743 server_base.cc:1034] running on GCE node
I20260923 22:19:42.158384 18743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:42.158658 18743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:42.159802 18743 hybrid_clock.cc:648] HybridClock initialized: now 1790201982159801 us; error 35 us; skew 500 ppm
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:42.163185 18743 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:19:42.163895 18743 webserver.cc:460] Webserver started at http://127.4.207.1:34335/ using document root <none> and password file <none>
I20260923 22:19:42.164191 18743 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:42.164260 18743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:42.164393 18743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:42.165511 18743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "a220a71fd3074789aabab378ad8233ce"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "6da5a608c0c9a4fe3ae82049ba736538"
server_key_iv: "bd73a497630ecab5d5b85f43ba944e61"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.165879 18743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "a220a71fd3074789aabab378ad8233ce"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "6da5a608c0c9a4fe3ae82049ba736538"
server_key_iv: "bd73a497630ecab5d5b85f43ba944e61"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.167199 18743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:19:42.168015 18758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.168198 18743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.168321 18743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "a220a71fd3074789aabab378ad8233ce"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "6da5a608c0c9a4fe3ae82049ba736538"
server_key_iv: "bd73a497630ecab5d5b85f43ba944e61"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.168407 18743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:42.186898 18743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:42.189277 18743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:42.189391 18743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:42.189565 18743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:42.189843 18743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:42.189888 18743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.189922 18743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:42.189941 18743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.195533 18743 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:44725
I20260923 22:19:42.195567 18871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:44725 every 8 connection(s)
I20260923 22:19:42.195881 18743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:42.205302 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.196701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:37251 (local address 127.4.207.62:42257)
0923 22:19:42.196774 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.196777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.196872 (+ 95us) server_negotiation.cc:407] Connection header received
0923 22:19:42.197329 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.197330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.197341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.197419 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:42.198305 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.198867 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.200003 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.200211 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.200278 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.202286 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:42.202287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:42.202287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:42.202295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:42.204017 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.204405 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.204407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.204407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.204471 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.204756 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.204756 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.204757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.204871 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:42.204931 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.205154 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.205231 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:19:42.205435 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.196568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:37251)
0923 22:19:42.196740 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.196780 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:42.197243 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:42.197471 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:42.197474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:42.198027 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:42.198224 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.198229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.198953 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.198954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.199916 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.199919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.200291 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.200292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.200391 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.200874 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:42.200882 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:42.202194 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:42.204088 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:42.204094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:42.204096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:42.204314 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:42.204600 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:42.204600 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:42.204601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:42.204692 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:42.204930 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:42.204934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:42.205041 (+ 107us) client_negotiation.cc:769] Sending connection context
0923 22:19:42.205120 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 22:19:42.205242 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260923 22:19:42.205750 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 18743
I20260923 22:19:42.205874 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:19:42.206005 18872 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42257
I20260923 22:19:42.206064 4924 external_mini_cluster.cc:1428] Setting key 478f8c22eae38ed410c20a6390594f12
I20260923 22:19:42.206130 18872 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:42.206398 18872 heartbeater.cc:511] Master 127.4.207.62:42257 requested a full tablet report, sending...
I20260923 22:19:42.207018 18681 ts_manager.cc:194] Registered new tserver with Master: a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725)
I20260923 22:19:42.207705 18681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:37251
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:19:42.230350 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42257
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:42.310703 18877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:42.310896 18877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:42.310920 18877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:42.311972 18877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:42.312006 18877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:42.312029 18877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:19:42.313351 18877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:42.313546 18877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:42.313721 18877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:42.318624 18882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:42.318725 18877 server_base.cc:1034] running on GCE node
W20260923 22:19:42.318813 18885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.318850 18883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:42.319099 18877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:42.319347 18877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:42.320541 18877 hybrid_clock.cc:648] HybridClock initialized: now 1790201982320526 us; error 25 us; skew 500 ppm
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:42.322896 18877 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:19:42.323354 18877 webserver.cc:460] Webserver started at http://127.4.207.2:46633/ using document root <none> and password file <none>
I20260923 22:19:42.323594 18877 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:42.323663 18877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:42.323807 18877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:42.324880 18877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "9d149e4cc6c543798c8e6abc93f0ece3"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "e326e9365f0f4311c64545722757e963"
server_key_iv: "3e981777585d53a659d9134665abc92c"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.325237 18877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "9d149e4cc6c543798c8e6abc93f0ece3"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "e326e9365f0f4311c64545722757e963"
server_key_iv: "3e981777585d53a659d9134665abc92c"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.326644 18877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:19:42.327327 18892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.327551 18877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:19:42.327643 18877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "9d149e4cc6c543798c8e6abc93f0ece3"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "e326e9365f0f4311c64545722757e963"
server_key_iv: "3e981777585d53a659d9134665abc92c"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.327714 18877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:42.345304 18877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:42.347168 18877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:42.347337 18877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:42.347591 18877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:42.347894 18877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:42.347947 18877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.348001 18877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:42.348022 18877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.353754 18877 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:33529
I20260923 22:19:42.353793 19005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:33529 every 8 connection(s)
I20260923 22:19:42.354092 18877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:19:42.354882 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 18877
I20260923 22:19:42.355031 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:19:42.355222 4924 external_mini_cluster.cc:1428] Setting key c90cc31c7525693bec6f6f580d7dc349
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
I20260923 22:19:42.364637 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.354920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:47087 (local address 127.4.207.62:42257)
0923 22:19:42.354998 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.355001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.355226 (+ 225us) server_negotiation.cc:407] Connection header received
0923 22:19:42.356046 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.356048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.356061 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.356120 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:42.357106 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.357785 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.359101 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.359306 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.359391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.361555 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:42.361557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:42.361557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:42.361565 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:42.363051 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.363574 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.363575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.363575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.363633 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.363892 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.363892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.363893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.364029 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:42.364111 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.364455 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.364547 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 22:19:42.364817 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.354857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:47087)
0923 22:19:42.355060 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.355110 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:42.355860 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:42.356196 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:42.356202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:42.356797 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:42.356986 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.356991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.357878 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.357879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.358986 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.358989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.359399 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.359400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.359559 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.360094 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:42.360103 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:42.361453 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:42.363163 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:42.363170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:42.363172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:42.363490 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:42.363703 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:42.363704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:42.363704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:42.363819 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:42.364102 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:42.364108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:42.364318 (+ 210us) client_negotiation.cc:769] Sending connection context
0923 22:19:42.364420 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 22:19:42.364579 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260923 22:19:42.365307 19006 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42257
I20260923 22:19:42.365391 19006 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:42.365537 19006 heartbeater.cc:511] Master 127.4.207.62:42257 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
I20260923 22:19:42.365978 18681 ts_manager.cc:194] Registered new tserver with Master: 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529)
I20260923 22:19:42.366433 18681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:47087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:19:42.379680 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42257
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:42.460204 19011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:42.460400 19011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:42.460441 19011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:42.461580 19011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:42.461619 19011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:42.461644 19011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:19:42.463018 19011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:42.463279 19011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:42.463579 19011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:42.468286 19016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.468279 19019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.468333 19017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:42.468433 19011 server_base.cc:1034] running on GCE node
I20260923 22:19:42.468675 19011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:42.468888 19011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:42.470031 19011 hybrid_clock.cc:648] HybridClock initialized: now 1790201982470026 us; error 32 us; skew 500 ppm
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:42.472560 19011 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:19:42.473006 19011 webserver.cc:460] Webserver started at http://127.4.207.3:34939/ using document root <none> and password file <none>
I20260923 22:19:42.473235 19011 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:42.473299 19011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:42.473448 19011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:19:42.474498 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "b9c595adacf24306b5742e7d15bfd6cf"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "38163d67227f18193b841ce7ef8ab32a"
server_key_iv: "dbb2145989a526714eb5177ae7170425"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.474850 19011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "b9c595adacf24306b5742e7d15bfd6cf"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "38163d67227f18193b841ce7ef8ab32a"
server_key_iv: "dbb2145989a526714eb5177ae7170425"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.476212 19011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:19:42.476925 19026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.477090 19011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:19:42.477198 19011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "b9c595adacf24306b5742e7d15bfd6cf"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "38163d67227f18193b841ce7ef8ab32a"
server_key_iv: "dbb2145989a526714eb5177ae7170425"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.477274 19011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:42.496750 19011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:42.498950 19011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:42.499099 19011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:42.499316 19011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:42.499651 19011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:42.499699 19011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.499737 19011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:42.499761 19011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.505295 19011 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:42077
I20260923 22:19:42.505332 19139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:42077 every 8 connection(s)
I20260923 22:19:42.505625 19011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:42.514271 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 19011
I20260923 22:19:42.514360 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:19:42.514416 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.506321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:35689 (local address 127.4.207.62:42257)
0923 22:19:42.506395 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.506397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.506579 (+ 182us) server_negotiation.cc:407] Connection header received
0923 22:19:42.507043 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.507044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.507054 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.507110 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:42.508147 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.508657 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.509581 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.509755 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.509807 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.511919 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:42.511920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:42.511921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:42.511927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:42.513162 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.513498 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.513499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.513499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.513570 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.513782 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.513783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.513783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.513878 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:42.513925 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.514220 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.514310 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:19:42.514441 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.506290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:35689)
0923 22:19:42.506452 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.506485 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:42.506924 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:42.507171 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:42.507175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:42.507862 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:42.508053 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.508058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.508730 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.508730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.509501 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.509503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.509825 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.509826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.509915 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.510410 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:42.510434 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:42.511829 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:42.513228 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:42.513233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:42.513235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:42.513435 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:42.513638 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:42.513638 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:42.513639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:42.513729 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:42.513929 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:42.513932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:42.514053 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 22:19:42.514133 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 22:19:42.514244 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":97,"threads_started":1}
I20260923 22:19:42.514559 4924 external_mini_cluster.cc:1428] Setting key 123c174d0855323311ae36cdc5a09900
I20260923 22:19:42.515123 19140 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42257
I20260923 22:19:42.515219 19140 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:42.515401 19140 heartbeater.cc:511] Master 127.4.207.62:42257 requested a full tablet report, sending...
I20260923 22:19:42.515802 18681 ts_manager.cc:194] Registered new tserver with Master: b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077)
I20260923 22:19:42.516170 18681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:35689
I20260923 22:19:42.526494 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:42.533893 18722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.527929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.4.207.62:42257)
0923 22:19:42.528065 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.528067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.528218 (+ 151us) server_negotiation.cc:407] Connection header received
0923 22:19:42.528254 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.528255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.528264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.528316 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:42.528822 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.529270 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.529962 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.530146 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.530243 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.531627 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:42.531628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:42.531628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:42.531634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:42.532772 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.533084 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.533085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.533085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.533138 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:42.533349 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:42.533350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:42.533350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:42.533497 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:42.533614 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.533703 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.533763 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260923 22:19:42.535529 18681 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:19:42.537077 18681 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:19:42.543164 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.539200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33529 (local address 127.0.0.1:45842)
0923 22:19:42.539524 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.539535 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:42.539590 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:42.539892 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:42.539893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:42.539895 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:42.539931 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.539932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.540966 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.540967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.541997 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.541999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.542863 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.542864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.542942 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.542957 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 22:19:42.543031 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:19:42.543089 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":66,"threads_started":1}
I20260923 22:19:42.543180 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.539140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:44725 (local address 127.0.0.1:45632)
0923 22:19:42.539461 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.539504 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:42.539566 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:42.539806 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:42.539809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:42.539811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:42.539874 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.539876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.541244 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.541246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.542299 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.542301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.542970 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.542970 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.543040 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.543042 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:19:42.543097 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:19:42.543126 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":59,"threads_started":1}
I20260923 22:19:42.543365 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.539302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.4.207.1:44725)
0923 22:19:42.539556 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.539561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.539573 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:19:42.539673 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.539677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.539695 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.539793 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:42.539932 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.541129 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.542375 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.542872 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.542965 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.543017 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.543118 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.543152 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":87,"threads_started":1}
I20260923 22:19:42.543571 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.539349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.4.207.2:33529)
0923 22:19:42.539650 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.539654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.539662 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:19:42.539697 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.539699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.539712 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.539814 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:42.540022 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.540868 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.542093 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.542807 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.542866 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.542923 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.543391 (+ 468us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.543468 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":75,"threads_started":1}
I20260923 22:19:42.543944 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.539420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.4.207.3:42077)
0923 22:19:42.539734 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.539739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.540194 (+ 455us) server_negotiation.cc:407] Connection header received
0923 22:19:42.540234 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.540237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.540251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.540327 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:42.540445 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.541508 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.542786 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.543531 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.543600 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.543666 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.543834 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.543869 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":99,"threads_started":1}
I20260923 22:19:42.544090 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.539334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:42077 (local address 127.0.0.1:57170)
0923 22:19:42.540028 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.540037 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:42.540086 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:42.540342 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:42.540343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:42.540345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:42.540386 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.540387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.541671 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.541672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.542697 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.542699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.543600 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.543601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.543729 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.543733 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:42.543818 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 22:19:42.543859 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":8,"thread_start_us":46,"threads_started":1}
I20260923 22:19:42.544215 18940 tablet_service.cc:1467] Processing CreateTablet for tablet 40a230601df54c549e1d03704ce03d6a (DEFAULT_TABLE table=test-table [id=b92a7f58ea5a4612bdd7fc497d18333f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:42.544214 18806 tablet_service.cc:1467] Processing CreateTablet for tablet 40a230601df54c549e1d03704ce03d6a (DEFAULT_TABLE table=test-table [id=b92a7f58ea5a4612bdd7fc497d18333f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:42.544560 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a230601df54c549e1d03704ce03d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:42.544560 18940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a230601df54c549e1d03704ce03d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:42.547905 19154 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Bootstrap starting.
I20260923 22:19:42.548527 19154 tablet_bootstrap.cc:654] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:42.548498 19074 tablet_service.cc:1467] Processing CreateTablet for tablet 40a230601df54c549e1d03704ce03d6a (DEFAULT_TABLE table=test-table [id=b92a7f58ea5a4612bdd7fc497d18333f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:42.548830 19074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a230601df54c549e1d03704ce03d6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:42.549100 19154 log.cc:824] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:42.549422 19155 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Bootstrap starting.
I20260923 22:19:42.549927 19154 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: No bootstrap required, opened a new log
I20260923 22:19:42.550000 19154 ts_tablet_manager.cc:1397] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:42.550127 19155 tablet_bootstrap.cc:654] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:42.550751 19155 log.cc:824] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:42.551532 19155 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: No bootstrap required, opened a new log
I20260923 22:19:42.551614 19155 ts_tablet_manager.cc:1397] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:42.551925 19158 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: Bootstrap starting.
I20260923 22:19:42.552186 19154 raft_consensus.cc:348] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.552363 19154 raft_consensus.cc:374] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:42.552400 19154 raft_consensus.cc:729] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a220a71fd3074789aabab378ad8233ce, State: Initialized, Role: FOLLOWER
I20260923 22:19:42.552563 19158 tablet_bootstrap.cc:654] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:42.552536 19154 consensus_queue.cc:260] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.552781 19154 ts_tablet_manager.cc:1428] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:19:42.552807 18872 heartbeater.cc:503] Master 127.4.207.62:42257 was elected leader, sending a full tablet report...
I20260923 22:19:42.553141 19158 log.cc:824] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:42.553556 19155 raft_consensus.cc:348] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.553764 19155 raft_consensus.cc:374] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:42.553809 19155 raft_consensus.cc:729] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d149e4cc6c543798c8e6abc93f0ece3, State: Initialized, Role: FOLLOWER
I20260923 22:19:42.553925 19155 consensus_queue.cc:260] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.553990 19158 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: No bootstrap required, opened a new log
I20260923 22:19:42.554059 19158 ts_tablet_manager.cc:1397] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:42.554190 19155 ts_tablet_manager.cc:1428] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:19:42.554252 19006 heartbeater.cc:503] Master 127.4.207.62:42257 was elected leader, sending a full tablet report...
I20260923 22:19:42.556015 19158 raft_consensus.cc:348] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.556144 19158 raft_consensus.cc:374] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:42.556166 19158 raft_consensus.cc:729] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c595adacf24306b5742e7d15bfd6cf, State: Initialized, Role: FOLLOWER
I20260923 22:19:42.556267 19158 consensus_queue.cc:260] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.556541 19158 ts_tablet_manager.cc:1428] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:42.556560 19140 heartbeater.cc:503] Master 127.4.207.62:42257 was elected leader, sending a full tablet report...
W20260923 22:19:42.604655 19007 tablet.cc:2369] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:42.678939 19162 raft_consensus.cc:482] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:42.679068 19162 raft_consensus.cc:504] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.679546 19162 leader_election.cc:288] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725), 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529)
I20260923 22:19:42.683084 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.679879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:34693 (local address 127.4.207.1:44725)
0923 22:19:42.679932 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.679935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.680104 (+ 169us) server_negotiation.cc:407] Connection header received
0923 22:19:42.680186 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.680187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.680199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.680276 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:42.680429 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.680902 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.682166 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.682729 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.682809 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.682819 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.682971 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.683014 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 22:19:42.683111 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.679871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:51989 (local address 127.4.207.2:33529)
0923 22:19:42.679951 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.679953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.679962 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:42.679993 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.679993 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.680004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.680069 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:42.680235 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.680718 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.682138 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.682746 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.682809 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.682819 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.682998 (+ 179us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.683039 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 22:19:42.683099 19163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.679778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:44725 (local address 127.4.207.3:34693)
0923 22:19:42.680016 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.680033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:42.680118 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:42.680278 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:42.680279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:42.680280 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:42.680360 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.680362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.680997 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.680998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.682076 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.682078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.682813 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.682815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.682891 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.682895 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:19:42.682951 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 22:19:42.682994 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":71,"thread_start_us":113,"threads_started":1}
I20260923 22:19:42.683123 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.679771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33529 (local address 127.4.207.3:51989)
0923 22:19:42.679848 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.679863 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:42.679929 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:42.680071 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:42.680072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:42.680073 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:42.680152 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.680154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.680810 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.680811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.682028 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:42.682031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.682853 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.682854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:42.682938 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.682941 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:19:42.682993 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:19:42.683043 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":31}
I20260923 22:19:42.683840 18960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a230601df54c549e1d03704ce03d6a" candidate_uuid: "b9c595adacf24306b5742e7d15bfd6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" is_pre_election: true
I20260923 22:19:42.683835 18826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a230601df54c549e1d03704ce03d6a" candidate_uuid: "b9c595adacf24306b5742e7d15bfd6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a220a71fd3074789aabab378ad8233ce" is_pre_election: true
I20260923 22:19:42.683987 18960 raft_consensus.cc:2454] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c595adacf24306b5742e7d15bfd6cf in term 0.
I20260923 22:19:42.683995 18826 raft_consensus.cc:2454] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c595adacf24306b5742e7d15bfd6cf in term 0.
I20260923 22:19:42.684231 19030 leader_election.cc:302] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d149e4cc6c543798c8e6abc93f0ece3, b9c595adacf24306b5742e7d15bfd6cf; no voters:
I20260923 22:19:42.684392 19162 raft_consensus.cc:2781] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:42.684455 19162 raft_consensus.cc:482] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:42.684502 19162 raft_consensus.cc:3037] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:42.685196 19162 raft_consensus.cc:504] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.685334 19162 leader_election.cc:288] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [CANDIDATE]: Term 1 election: Requested vote from peers a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725), 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529)
I20260923 22:19:42.685472 18826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a230601df54c549e1d03704ce03d6a" candidate_uuid: "b9c595adacf24306b5742e7d15bfd6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a220a71fd3074789aabab378ad8233ce"
I20260923 22:19:42.685487 18960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a230601df54c549e1d03704ce03d6a" candidate_uuid: "b9c595adacf24306b5742e7d15bfd6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d149e4cc6c543798c8e6abc93f0ece3"
I20260923 22:19:42.685562 18826 raft_consensus.cc:3037] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:42.685564 18960 raft_consensus.cc:3037] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:42.686254 18960 raft_consensus.cc:2454] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c595adacf24306b5742e7d15bfd6cf in term 1.
I20260923 22:19:42.686348 18826 raft_consensus.cc:2454] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c595adacf24306b5742e7d15bfd6cf in term 1.
I20260923 22:19:42.686432 19030 leader_election.cc:302] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d149e4cc6c543798c8e6abc93f0ece3, b9c595adacf24306b5742e7d15bfd6cf; no voters:
I20260923 22:19:42.686529 19162 raft_consensus.cc:2781] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:42.686668 19162 raft_consensus.cc:686] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 1 LEADER]: Becoming Leader. State: Replica: b9c595adacf24306b5742e7d15bfd6cf, State: Running, Role: LEADER
I20260923 22:19:42.686760 19162 consensus_queue.cc:237] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:42.687660 18680 catalog_manager.cc:5495] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3). New cstate: current_term: 1 leader_uuid: "b9c595adacf24306b5742e7d15bfd6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } health_report { overall_health: UNKNOWN } } }
W20260923 22:19:42.696533 18873 tablet.cc:2369] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:42.723227 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.721017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.4.207.3:42077)
0923 22:19:42.721073 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:42.721075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:42.721082 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:19:42.721118 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:42.721119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:42.721139 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:42.721199 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:42.721307 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.721770 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.722480 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:42.722666 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:42.722713 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:42.722843 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:42.722939 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:42.722997 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:42.723078 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 22:19:42.723093 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:19:42.725126 18826 raft_consensus.cc:1261] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Refusing update from remote peer b9c595adacf24306b5742e7d15bfd6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:42.725132 18960 raft_consensus.cc:1261] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 1 FOLLOWER]: Refusing update from remote peer b9c595adacf24306b5742e7d15bfd6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:42.725487 19164 consensus_queue.cc:1035] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:42.725623 19162 consensus_queue.cc:1035] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:42.728718 19169 mvcc.cc:205] Tried to move back new op lower bound from 7332667321239248896 to 7332667321085816832. Current Snapshot: MvccSnapshot[applied={T|T < 7332667321239248896 or (T in {7332667321239248896})}]
I20260923 22:19:42.728747 19167 mvcc.cc:205] Tried to move back new op lower bound from 7332667321239248896 to 7332667321085816832. Current Snapshot: MvccSnapshot[applied={T|T < 7332667321239248896}]
I20260923 22:19:42.730787 18681 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:42.730895 18681 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:42.731637 18681 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Sending DeleteTablet for 3 replicas of tablet 40a230601df54c549e1d03704ce03d6a
I20260923 22:19:42.731950 18806 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a230601df54c549e1d03704ce03d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:42 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:45632
I20260923 22:19:42.732029 19074 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a230601df54c549e1d03704ce03d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:42 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:57170
I20260923 22:19:42.732249 19177 tablet_replica.cc:330] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: stopping tablet replica
I20260923 22:19:42.732262 18940 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a230601df54c549e1d03704ce03d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:42 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:45842
I20260923 22:19:42.732362 19177 raft_consensus.cc:2229] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:42.732470 19177 raft_consensus.cc:2258] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:42.732465 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 18743
I20260923 22:19:42.735563 19179 tablet_replica.cc:330] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: stopping tablet replica
I20260923 22:19:42.735706 19179 raft_consensus.cc:2229] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:42.735816 19179 raft_consensus.cc:2258] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:42.735992 19178 tablet_replica.cc:330] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: stopping tablet replica
I20260923 22:19:42.736076 19178 raft_consensus.cc:2229] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:19:42.736209 19178 raft_consensus.cc:2258] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:42.736406 19179 ts_tablet_manager.cc:1905] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:42.736564 19178 ts_tablet_manager.cc:1905] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:42.738745 19178 ts_tablet_manager.cc:1918] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:42.738828 19178 log.cc:1196] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/40a230601df54c549e1d03704ce03d6a
W20260923 22:19:42.738883 18665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:44725 (error 108)
I20260923 22:19:42.739028 19179 ts_tablet_manager.cc:1918] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:42.739118 19178 ts_tablet_manager.cc:1939] T 40a230601df54c549e1d03704ce03d6a P b9c595adacf24306b5742e7d15bfd6cf: Deleting consensus metadata
I20260923 22:19:42.739212 19179 log.cc:1196] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/40a230601df54c549e1d03704ce03d6a
I20260923 22:19:42.739411 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 18877
I20260923 22:19:42.739528 18665 catalog_manager.cc:4841] TS b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077): tablet 40a230601df54c549e1d03704ce03d6a (table test-table [id=b92a7f58ea5a4612bdd7fc497d18333f]) successfully deleted
I20260923 22:19:42.739805 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.739685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:44725 (local address 127.0.0.1:45644)
0923 22:19:42.739733 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.739768 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 22:19:42.739923 18665 catalog_manager.cc:4584] TS a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 40a230601df54c549e1d03704ce03d6a: Network error: Client connection negotiation failed: client connection to 127.4.207.1:44725: connect: Connection refused (error 111)
I20260923 22:19:42.744036 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 19011
I20260923 22:19:42.744801 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:42.744182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33529 (local address 127.0.0.1:45856)
0923 22:19:42.744733 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:42.744763 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.2:33529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529}
I20260923 22:19:42.748687 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 18649
I20260923 22:19:42.753890 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42257
--webserver_interface=127.4.207.62
--webserver_port=38217
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:42.833729 19181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:42.833913 19181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:42.833935 19181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:42.835045 19181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:42.835084 19181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:42.835098 19181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:42.835161 19181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:42.835179 19181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:42.836588 19181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=38217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:42.836819 19181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:42.837073 19181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:42.845299 19187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.845336 19188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.845330 19190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:42.845477 19181 server_base.cc:1034] running on GCE node
I20260923 22:19:42.845808 19181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:42.846159 19181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:42.847339 19181 hybrid_clock.cc:648] HybridClock initialized: now 1790201982847329 us; error 29 us; skew 500 ppm
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:42.853305 19181 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:19:42.853747 19181 webserver.cc:460] Webserver started at http://127.4.207.62:38217/ using document root <none> and password file <none>
I20260923 22:19:42.853989 19181 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:42.854069 19181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:42.855423 19181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:42.856171 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.856369 19181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:42.856474 19181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "88f233cbd5f44b8fb81bcebe3b5b388c"
format_stamp: "Formatted at 2026-09-23 22:19:41 on dist-test-slave-7511"
server_key: "80c6ae96b3dbb1deaa3f8f934bbe514c"
server_key_iv: "cfda1236979e65647546dee16c720c1e"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.856896 19181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:42.868844 19181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:42.870841 19181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:42.871006 19181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:42.874852 19181 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:42257
I20260923 22:19:42.874881 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:42257 every 8 connection(s)
I20260923 22:19:42.875257 19181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:42.876842 19250 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:19:42.877471 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Bootstrap starting.
I20260923 22:19:42.878759 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 19181
I20260923 22:19:42.879051 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:44725
--local_ip_for_outbound_sockets=127.4.207.1
--tserver_master_addrs=127.4.207.62:42257
--webserver_port=34335
--webserver_interface=127.4.207.1
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:42.886334 19250 log.cc:824] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:42.888360 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:42.888656 19250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Bootstrap complete.
I20260923 22:19:42.890828 19250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:42.891052 19250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f233cbd5f44b8fb81bcebe3b5b388c, State: Initialized, Role: FOLLOWER
I20260923 22:19:42.891222 19250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:42.891283 19250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:42.891330 19250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:42.891378 19250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:19:42.892407 19250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:42.892518 19250 leader_election.cc:302] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f233cbd5f44b8fb81bcebe3b5b388c; no voters:
I20260923 22:19:42.892709 19250 leader_election.cc:288] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 22:19:42.892742 19254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:19:42.892841 19254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 2 LEADER]: Becoming Leader. State: Replica: 88f233cbd5f44b8fb81bcebe3b5b388c, State: Running, Role: LEADER
I20260923 22:19:42.892930 19254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:42.892961 19250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:42.893167 19255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } } }
I20260923 22:19:42.893263 19255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:42.893535 19258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:42.893906 19256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f233cbd5f44b8fb81bcebe3b5b388c. Latest consensus state: current_term: 2 leader_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } } }
I20260923 22:19:42.894080 19256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:42.894426 19258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:42.894541 19258 catalog_manager.cc:1259] Loaded cluster ID: 3db2b12ec87749c2a377e033d971a7f5
I20260923 22:19:42.894577 19258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:19:42.894795 19258 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 22:19:42.899487 19258 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 22:19:42.901916 19258 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:19:42.902309 19258 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Loaded TSK: 0
I20260923 22:19:42.902539 19258 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:19:42.960621 19252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:42.960844 19252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:42.960878 19252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:42.962033 19252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:42.962092 19252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:42.962136 19252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:19:42.963735 19252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:44725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=34335
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:42.964002 19252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:42.964229 19252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:42.969156 19280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.969234 19277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:42.969164 19278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:42.969610 19252 server_base.cc:1034] running on GCE node
I20260923 22:19:42.969813 19252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:42.970057 19252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:42.971249 19252 hybrid_clock.cc:648] HybridClock initialized: now 1790201982971236 us; error 32 us; skew 500 ppm
Sep 23 22:19:42 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:42.974294 19252 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:19:42.974988 19252 webserver.cc:460] Webserver started at http://127.4.207.1:34335/ using document root <none> and password file <none>
I20260923 22:19:42.975284 19252 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:42.975375 19252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:42.977351 19252 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:42.978286 19287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:42.978456 19252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:42.978559 19252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "a220a71fd3074789aabab378ad8233ce"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "6da5a608c0c9a4fe3ae82049ba736538"
server_key_iv: "bd73a497630ecab5d5b85f43ba944e61"
server_key_version: "encryptionkey@0"
I20260923 22:19:42.978945 19252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:42.988821 19252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:42.991139 19252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:42.991326 19252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:42.991593 19252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:42.992057 19294 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:19:42.994331 19252 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:19:42.994411 19252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260923 22:19:42.994459 19252 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:19:42.995126 19252 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:19:42.995184 19252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:42.995245 19294 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Bootstrap starting.
I20260923 22:19:43.002249 19252 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:44725
I20260923 22:19:43.002624 19252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:43.004381 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 19252
I20260923 22:19:43.004678 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:33529
--local_ip_for_outbound_sockets=127.4.207.2
--tserver_master_addrs=127.4.207.62:42257
--webserver_port=46633
--webserver_interface=127.4.207.2
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:43.005888 19294 log.cc:824] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:43.007669 19401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:44725 every 8 connection(s)
I20260923 22:19:43.008661 19294 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:43.008970 19294 tablet_bootstrap.cc:492] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Bootstrap complete.
I20260923 22:19:43.010149 19294 ts_tablet_manager.cc:1397] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260923 22:19:43.011931 19294 raft_consensus.cc:348] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:43.012279 19294 raft_consensus.cc:729] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a220a71fd3074789aabab378ad8233ce, State: Initialized, Role: FOLLOWER
I20260923 22:19:43.012612 19294 consensus_queue.cc:260] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } }
I20260923 22:19:43.012899 19294 ts_tablet_manager.cc:1428] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Sep 23 22:19:43 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201982, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:43.024439 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.003400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:58935)
0923 22:19:43.007967 (+ 4567us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.008010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:43.012417 (+ 4407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:43.013236 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:43.013240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:43.013667 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:43.013982 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.013990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.015625 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.015626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.016872 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.016875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.017728 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.017729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.017861 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.018520 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:43.018530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:43.020366 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:43.022725 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:43.022731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:43.022733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:43.023018 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:43.023292 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:43.023293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:43.023293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:43.023408 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:43.023866 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:43.023871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:43.024021 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 22:19:43.024111 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 22:19:43.024255 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4514,"thread_start_us":116,"threads_started":1}
I20260923 22:19:43.024771 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.003519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:58935 (local address 127.4.207.62:42257)
0923 22:19:43.008275 (+ 4756us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.008284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.008328 (+ 44us) server_negotiation.cc:407] Connection header received
0923 22:19:43.012682 (+ 4354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.012690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.012835 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.013088 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:43.014131 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.015513 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.016973 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.017613 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.017749 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.020497 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:43.020501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:43.020501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:43.020514 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:43.022630 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:43.023101 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:43.023104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:43.023104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:43.023211 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:43.023520 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:43.023520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:43.023521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:43.023796 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:43.023873 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.024197 (+ 324us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.024319 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4705,"thread_start_us":67,"threads_started":1}
I20260923 22:19:43.025564 19402 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42257
I20260923 22:19:43.025673 19402 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:43.025832 19402 heartbeater.cc:511] Master 127.4.207.62:42257 requested a full tablet report, sending...
I20260923 22:19:43.026553 19214 ts_manager.cc:194] Registered new tserver with Master: a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725)
I20260923 22:19:43.027724 19214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:58935
I20260923 22:19:43.028767 19402 heartbeater.cc:503] Master 127.4.207.62:42257 was elected leader, sending a full tablet report...
I20260923 22:19:43.035414 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.027652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:44725 (local address 127.0.0.1:45646)
0923 22:19:43.028131 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.028164 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:43.028236 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:43.031743 (+ 3507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:43.031747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:43.031750 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:43.031823 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.031826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.033012 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.033013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.034252 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.034254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.035133 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.035134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.035212 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.035224 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 22:19:43.035290 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:19:43.035337 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":115,"threads_started":1}
I20260923 22:19:43.035415 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.027927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.4.207.1:44725)
0923 22:19:43.031558 (+ 3631us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.031563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.031573 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:19:43.031606 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.031608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.031624 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.031716 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:43.031903 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.032908 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.034340 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.035050 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.035124 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.035173 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.035313 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.035350 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3600,"thread_start_us":79,"threads_started":1}
I20260923 22:19:43.035964 19321 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a230601df54c549e1d03704ce03d6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:42 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:45646
I20260923 22:19:43.036509 19411 tablet_replica.cc:330] stopping tablet replica
I20260923 22:19:43.036619 19411 raft_consensus.cc:2229] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:43.036680 19411 raft_consensus.cc:2258] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:43.037019 19411 ts_tablet_manager.cc:1905] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:43.039633 19411 ts_tablet_manager.cc:1918] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:43.039702 19411 log.cc:1196] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/40a230601df54c549e1d03704ce03d6a
I20260923 22:19:43.039911 19411 ts_tablet_manager.cc:1939] T 40a230601df54c549e1d03704ce03d6a P a220a71fd3074789aabab378ad8233ce: Deleting consensus metadata
I20260923 22:19:43.040395 19198 catalog_manager.cc:4841] TS a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725): tablet 40a230601df54c549e1d03704ce03d6a (table test-table [id=b92a7f58ea5a4612bdd7fc497d18333f]) successfully deleted
W20260923 22:19:43.094559 19406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:43.094794 19406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:43.094834 19406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:43.096068 19406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:43.096135 19406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:43.096184 19406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:19:43.097649 19406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=46633
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:43.097911 19406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:43.098152 19406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:43.103209 19419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:43.103232 19417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:43.103217 19416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:43.103612 19406 server_base.cc:1034] running on GCE node
I20260923 22:19:43.103816 19406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:43.104038 19406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:43.105189 19406 hybrid_clock.cc:648] HybridClock initialized: now 1790201983105170 us; error 32 us; skew 500 ppm
Sep 23 22:19:43 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:43.107697 19406 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:19:43.108119 19406 webserver.cc:460] Webserver started at http://127.4.207.2:46633/ using document root <none> and password file <none>
I20260923 22:19:43.108345 19406 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:43.108420 19406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:43.109741 19406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:43.110512 19426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.110693 19406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:43.110790 19406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "9d149e4cc6c543798c8e6abc93f0ece3"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "e326e9365f0f4311c64545722757e963"
server_key_iv: "3e981777585d53a659d9134665abc92c"
server_key_version: "encryptionkey@0"
I20260923 22:19:43.111145 19406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:43.139220 19406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:43.141732 19406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:43.141887 19406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:43.142136 19406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:43.142627 19433 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260923 22:19:43.144147 19433 ts_tablet_manager.cc:1852] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260923 22:19:43.144407 19433 ts_tablet_manager.cc:1905] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:43.146700 19433 ts_tablet_manager.cc:1918] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:43.146812 19433 log.cc:1196] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/40a230601df54c549e1d03704ce03d6a
I20260923 22:19:43.147095 19433 ts_tablet_manager.cc:1939] T 40a230601df54c549e1d03704ce03d6a P 9d149e4cc6c543798c8e6abc93f0ece3: Deleting consensus metadata
I20260923 22:19:43.147326 19406 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260923 22:19:43.147513 19406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260923 22:19:43.147614 19406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:43.147647 19406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.153682 19406 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:33529
I20260923 22:19:43.153734 19540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:33529 every 8 connection(s)
I20260923 22:19:43.154043 19406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:19:43 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:43.160858 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 19406
I20260923 22:19:43.161155 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:42077
--local_ip_for_outbound_sockets=127.4.207.3
--tserver_master_addrs=127.4.207.62:42257
--webserver_port=34939
--webserver_interface=127.4.207.3
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:43.163538 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.154776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:33817 (local address 127.4.207.62:42257)
0923 22:19:43.154843 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.154845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.155073 (+ 228us) server_negotiation.cc:407] Connection header received
0923 22:19:43.155635 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.155637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.155651 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.155706 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:43.156311 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.156811 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.157825 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.158067 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.158124 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.160484 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:43.160485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:43.160486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:43.160493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:43.161961 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:43.162488 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:43.162489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:43.162489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:43.162542 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:43.162839 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:43.162840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:43.162840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:43.162940 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:43.162989 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.163340 (+ 351us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.163414 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:19:43.163815 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.154790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:33817)
0923 22:19:43.154945 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.154980 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:43.155531 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:43.155759 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:43.155763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:43.156016 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:43.156224 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.156228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.156897 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.156898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.157735 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.157737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.158142 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.158142 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.158238 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.158814 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:43.158822 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:43.160366 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:43.162087 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:43.162093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:43.162096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:43.162404 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:43.162635 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:43.162635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:43.162636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:43.162776 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:43.163002 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:43.163007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:43.163188 (+ 181us) client_negotiation.cc:769] Sending connection context
0923 22:19:43.163313 (+ 125us) client_negotiation.cc:240] Negotiation successful
0923 22:19:43.163508 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260923 22:19:43.164333 19541 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42257
I20260923 22:19:43.164469 19541 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:43.164691 19541 heartbeater.cc:511] Master 127.4.207.62:42257 requested a full tablet report, sending...
I20260923 22:19:43.165143 19214 ts_manager.cc:194] Registered new tserver with Master: 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529)
I20260923 22:19:43.165647 19214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:33817
W20260923 22:19:43.242249 19544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:43.242455 19544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:43.242491 19544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:43.243731 19544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:43.243795 19544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:43.243841 19544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:19:43.245308 19544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=34939
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:43.245582 19544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:43.245821 19544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:43.250864 19549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:43.250869 19550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:43.250927 19552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:43.251066 19544 server_base.cc:1034] running on GCE node
I20260923 22:19:43.251403 19544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:43.251713 19544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:43.252880 19544 hybrid_clock.cc:648] HybridClock initialized: now 1790201983252876 us; error 35 us; skew 500 ppm
Sep 23 22:19:43 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:43.255394 19544 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:19:43.255941 19544 webserver.cc:460] Webserver started at http://127.4.207.3:34939/ using document root <none> and password file <none>
I20260923 22:19:43.256181 19544 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:43.256265 19544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:43.257673 19544 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:43.258320 19559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.258489 19544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:19:43.258615 19544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "b9c595adacf24306b5742e7d15bfd6cf"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "38163d67227f18193b841ce7ef8ab32a"
server_key_iv: "dbb2145989a526714eb5177ae7170425"
server_key_version: "encryptionkey@0"
I20260923 22:19:43.258953 19544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:43.292721 19544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:43.294826 19544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:43.294981 19544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:43.295212 19544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:43.295586 19544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:43.295639 19544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.295696 19544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:43.295723 19544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.301574 19544 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:42077
I20260923 22:19:43.301608 19672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:42077 every 8 connection(s)
I20260923 22:19:43.301954 19544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:19:43.307333 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 19544
Sep 23 22:19:43 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:19:43.311594 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.302616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:41603 (local address 127.4.207.62:42257)
0923 22:19:43.302680 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.302682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.302904 (+ 222us) server_negotiation.cc:407] Connection header received
0923 22:19:43.303385 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.303386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.303396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.303490 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:43.304411 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.304861 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.305808 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.305978 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.306025 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.308364 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:43.308365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:43.308366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:43.308374 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:43.309908 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:43.310280 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:43.310281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:43.310282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:43.310341 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:43.310683 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:43.310685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:43.310685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:43.310829 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:43.310920 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.311365 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.311463 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:19:43.311661 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.302633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:41603)
0923 22:19:43.302773 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.302825 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:43.303297 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:43.303559 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:43.303563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:43.304123 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:43.304324 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.304329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.304942 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.304943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.305737 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.305739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.306046 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.306047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.306142 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.306665 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:43.306673 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:43.308246 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:43.309987 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:43.309993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:43.309994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:43.310210 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:43.310409 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:43.310410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:43.310410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:43.310559 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:43.311023 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:43.311030 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:43.311205 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 22:19:43.311313 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 22:19:43.311427 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260923 22:19:43.312201 19673 heartbeater.cc:347] Connected to a master server at 127.4.207.62:42257
I20260923 22:19:43.312301 19673 heartbeater.cc:464] Registering TS with master...
I20260923 22:19:43.312486 19673 heartbeater.cc:511] Master 127.4.207.62:42257 requested a full tablet report, sending...
I20260923 22:19:43.312902 19214 ts_manager.cc:194] Registered new tserver with Master: b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077)
I20260923 22:19:43.313262 19214 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:41603
I20260923 22:19:43.314791 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.307794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.4.207.62:42257)
0923 22:19:43.307940 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.307942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.307951 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:43.307993 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.307994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.308004 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.308072 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:43.308720 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.309339 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.310317 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.310613 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.310679 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.312177 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:43.312178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:43.312178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:43.312185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:43.313938 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:43.314237 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:43.314238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:43.314238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:43.314287 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:43.314400 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:43.314400 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:43.314400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:43.314494 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:43.314601 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.314660 (+ 59us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.314714 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260923 22:19:43.315526 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:19:43.317859 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.315849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54956 (local address 127.4.207.62:42257)
0923 22:19:43.315893 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.315895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.315969 (+ 74us) server_negotiation.cc:407] Connection header received
0923 22:19:43.316018 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.316019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.316045 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.316093 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:43.316161 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.316568 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.317220 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.317403 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.317467 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.317614 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:43.317692 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:43.317744 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.317773 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.317788 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:19:43.318106 19214 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:19:43.319836 19214 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:19:43.322166 19321 tablet_service.cc:1467] Processing CreateTablet for tablet 18478aaecd5d46c684d1e6f8b6d48a3e (DEFAULT_TABLE table=test-table [id=530aef9eb94e453aa3d69edb863e82cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:43.322376 19321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18478aaecd5d46c684d1e6f8b6d48a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:43.323899 19294 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Bootstrap starting.
I20260923 22:19:43.324201 19294 tablet_bootstrap.cc:654] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:43.324667 19294 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: No bootstrap required, opened a new log
I20260923 22:19:43.324712 19294 ts_tablet_manager.cc:1397] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:43.324811 19294 raft_consensus.cc:348] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.324880 19294 raft_consensus.cc:374] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:43.324924 19294 raft_consensus.cc:729] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a220a71fd3074789aabab378ad8233ce, State: Initialized, Role: FOLLOWER
I20260923 22:19:43.324973 19294 consensus_queue.cc:260] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.325078 19294 ts_tablet_manager.cc:1428] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.325610 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.321784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:42077 (local address 127.0.0.1:57180)
0923 22:19:43.321921 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.321931 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:43.321969 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:43.322379 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:43.322380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:43.322382 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:43.322414 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.322415 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.323538 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.323538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.324546 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.324548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.325400 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.325401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.325466 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.325471 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:19:43.325516 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 22:19:43.325557 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260923 22:19:43.325840 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.321855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.4.207.3:42077)
0923 22:19:43.322096 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.322099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.322107 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:19:43.322138 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.322139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.322152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.322228 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:43.322489 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.323427 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.324654 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.325325 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.325401 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.325453 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.325727 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.325770 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":102,"threads_started":1}
I20260923 22:19:43.326167 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.321723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33529 (local address 127.0.0.1:45862)
0923 22:19:43.321756 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.321766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:43.321804 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:43.322239 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:43.322240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:43.322241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:43.322296 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.322298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.323379 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.323380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.324469 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.324471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.325974 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.325974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.326027 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.326036 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 22:19:43.326079 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 22:19:43.326109 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260923 22:19:43.326215 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.321848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45862 (local address 127.4.207.2:33529)
0923 22:19:43.322061 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.322066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.322077 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:19:43.322122 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.322124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.322140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.322228 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:43.322360 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.323288 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.325151 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.325847 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.325897 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.325974 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.326106 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.326143 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260923 22:19:43.326599 19607 tablet_service.cc:1467] Processing CreateTablet for tablet 18478aaecd5d46c684d1e6f8b6d48a3e (DEFAULT_TABLE table=test-table [id=530aef9eb94e453aa3d69edb863e82cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:43.326643 19475 tablet_service.cc:1467] Processing CreateTablet for tablet 18478aaecd5d46c684d1e6f8b6d48a3e (DEFAULT_TABLE table=test-table [id=530aef9eb94e453aa3d69edb863e82cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:19:43.326982 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18478aaecd5d46c684d1e6f8b6d48a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:43.326998 19607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18478aaecd5d46c684d1e6f8b6d48a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:19:43.328780 19433 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Bootstrap starting.
I20260923 22:19:43.329473 19433 tablet_bootstrap.cc:654] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:43.329787 19433 log.cc:824] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:43.329898 19683 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Bootstrap starting.
I20260923 22:19:43.330545 19683 tablet_bootstrap.cc:654] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Neither blocks nor log segments found. Creating new log.
I20260923 22:19:43.330608 19433 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: No bootstrap required, opened a new log
I20260923 22:19:43.330668 19433 ts_tablet_manager.cc:1397] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:43.331046 19683 log.cc:824] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:43.331698 19683 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: No bootstrap required, opened a new log
I20260923 22:19:43.331751 19683 ts_tablet_manager.cc:1397] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:19:43.332247 19433 raft_consensus.cc:348] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.332396 19433 raft_consensus.cc:374] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:43.332442 19433 raft_consensus.cc:729] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d149e4cc6c543798c8e6abc93f0ece3, State: Initialized, Role: FOLLOWER
I20260923 22:19:43.332566 19433 consensus_queue.cc:260] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.332846 19433 ts_tablet_manager.cc:1428] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:43.332909 19541 heartbeater.cc:503] Master 127.4.207.62:42257 was elected leader, sending a full tablet report...
I20260923 22:19:43.333441 19683 raft_consensus.cc:348] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.333606 19683 raft_consensus.cc:374] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:19:43.333653 19683 raft_consensus.cc:729] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9c595adacf24306b5742e7d15bfd6cf, State: Initialized, Role: FOLLOWER
I20260923 22:19:43.333762 19683 consensus_queue.cc:260] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.334049 19683 ts_tablet_manager.cc:1428] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:19:43.334075 19673 heartbeater.cc:503] Master 127.4.207.62:42257 was elected leader, sending a full tablet report...
W20260923 22:19:43.404632 19542 tablet.cc:2369] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:19:43.508270 19403 tablet.cc:2369] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:19:43.552481 19674 tablet.cc:2369] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:19:43.652813 19687 raft_consensus.cc:482] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:43.652935 19687 raft_consensus.cc:504] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.653350 19687 leader_election.cc:288] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529), a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725)
I20260923 22:19:43.656618 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.653912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:52977 (local address 127.4.207.2:33529)
0923 22:19:43.653953 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.653955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.653964 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:19:43.654024 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.654025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.654035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.654124 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:43.654306 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.654884 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.655878 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.656241 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.656330 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.656340 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.656511 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.656549 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:19:43.656642 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.653581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33529 (local address 127.4.207.3:52977)
0923 22:19:43.653680 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.653693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:43.653751 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:43.654127 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:43.654128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:43.654130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:43.654206 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.654208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.654980 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.654981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.655782 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.655784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.656331 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.656332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.656420 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.656423 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:19:43.656502 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 22:19:43.656543 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260923 22:19:43.656675 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.653728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:47383 (local address 127.4.207.1:44725)
0923 22:19:43.653903 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.653906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.654162 (+ 256us) server_negotiation.cc:407] Connection header received
0923 22:19:43.654193 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.654194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.654205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.654267 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:19:43.654457 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.654957 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.655911 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.656281 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.656371 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.656385 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.656554 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.656600 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260923 22:19:43.656947 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.653628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:44725 (local address 127.4.207.3:47383)
0923 22:19:43.653885 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.653895 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:43.653950 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:43.654306 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:43.654307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:43.654310 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:19:43.654376 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.654378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.655056 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.655057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.655825 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:43.655827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.656374 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.656375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:43.656449 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.656454 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:19:43.656540 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 22:19:43.656580 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":188,"threads_started":1}
I20260923 22:19:43.656991 19495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18478aaecd5d46c684d1e6f8b6d48a3e" candidate_uuid: "b9c595adacf24306b5742e7d15bfd6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" is_pre_election: true
I20260923 22:19:43.657251 19495 raft_consensus.cc:2454] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c595adacf24306b5742e7d15bfd6cf in term 0.
I20260923 22:19:43.657284 19356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18478aaecd5d46c684d1e6f8b6d48a3e" candidate_uuid: "b9c595adacf24306b5742e7d15bfd6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a220a71fd3074789aabab378ad8233ce" is_pre_election: true
I20260923 22:19:43.657445 19356 raft_consensus.cc:2454] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9c595adacf24306b5742e7d15bfd6cf in term 0.
I20260923 22:19:43.657526 19563 leader_election.cc:302] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d149e4cc6c543798c8e6abc93f0ece3, b9c595adacf24306b5742e7d15bfd6cf; no voters:
I20260923 22:19:43.657645 19687 raft_consensus.cc:2781] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:19:43.657713 19687 raft_consensus.cc:482] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:19:43.657737 19687 raft_consensus.cc:3037] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:43.658468 19687 raft_consensus.cc:504] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.658627 19687 leader_election.cc:288] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [CANDIDATE]: Term 1 election: Requested vote from peers 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529), a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725)
I20260923 22:19:43.658771 19356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18478aaecd5d46c684d1e6f8b6d48a3e" candidate_uuid: "b9c595adacf24306b5742e7d15bfd6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a220a71fd3074789aabab378ad8233ce"
I20260923 22:19:43.658849 19356 raft_consensus.cc:3037] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:43.659667 19356 raft_consensus.cc:2454] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c595adacf24306b5742e7d15bfd6cf in term 1.
I20260923 22:19:43.659808 19495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18478aaecd5d46c684d1e6f8b6d48a3e" candidate_uuid: "b9c595adacf24306b5742e7d15bfd6cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d149e4cc6c543798c8e6abc93f0ece3"
I20260923 22:19:43.659865 19560 leader_election.cc:302] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a220a71fd3074789aabab378ad8233ce, b9c595adacf24306b5742e7d15bfd6cf; no voters:
I20260923 22:19:43.659886 19495 raft_consensus.cc:3037] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:19:43.659951 19687 raft_consensus.cc:2781] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:19:43.660112 19687 raft_consensus.cc:686] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 1 LEADER]: Becoming Leader. State: Replica: b9c595adacf24306b5742e7d15bfd6cf, State: Running, Role: LEADER
I20260923 22:19:43.660226 19687 consensus_queue.cc:237] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:43.660787 19495 raft_consensus.cc:2454] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9c595adacf24306b5742e7d15bfd6cf in term 1.
I20260923 22:19:43.661132 19214 catalog_manager.cc:5495] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf reported cstate change: term changed from 0 to 1, leader changed from <none> to b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3). New cstate: current_term: 1 leader_uuid: "b9c595adacf24306b5742e7d15bfd6cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } health_report { overall_health: UNKNOWN } } }
I20260923 22:19:43.677029 19681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.674882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57194 (local address 127.4.207.3:42077)
0923 22:19:43.674943 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:43.674944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:43.674952 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:19:43.674981 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:43.674982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:43.674999 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:43.675070 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:19:43.675174 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.675685 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.676350 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:43.676518 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:43.676572 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:43.676753 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:19:43.676842 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:19:43.676900 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:43.676939 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 22:19:43.676954 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:19:43.678748 19495 raft_consensus.cc:1261] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 1 FOLLOWER]: Refusing update from remote peer b9c595adacf24306b5742e7d15bfd6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:43.678747 19356 raft_consensus.cc:1261] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Refusing update from remote peer b9c595adacf24306b5742e7d15bfd6cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:19:43.679077 19687 consensus_queue.cc:1035] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:43.679229 19690 consensus_queue.cc:1035] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:19:43.681404 19694 mvcc.cc:205] Tried to move back new op lower bound from 7332667325145550848 to 7332667325072908288. Current Snapshot: MvccSnapshot[applied={T|T < 7332667325145550848}]
I20260923 22:19:43.682605 19693 mvcc.cc:205] Tried to move back new op lower bound from 7332667325145550848 to 7332667325072908288. Current Snapshot: MvccSnapshot[applied={T|T < 7332667325145550848 or (T in {7332667325145550848})}]
I20260923 22:19:43.683754 19213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:43.683830 19213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:19:43.684509 19213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Sending DeleteTablet for 3 replicas of tablet 18478aaecd5d46c684d1e6f8b6d48a3e
I20260923 22:19:43.684823 19321 tablet_service.cc:1514] Processing DeleteTablet for tablet 18478aaecd5d46c684d1e6f8b6d48a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:43 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:45646
I20260923 22:19:43.684839 19475 tablet_service.cc:1514] Processing DeleteTablet for tablet 18478aaecd5d46c684d1e6f8b6d48a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:43 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:45862
I20260923 22:19:43.685032 19703 tablet_replica.cc:330] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: stopping tablet replica
I20260923 22:19:43.685015 19607 tablet_service.cc:1514] Processing DeleteTablet for tablet 18478aaecd5d46c684d1e6f8b6d48a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:19:43 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:57180
I20260923 22:19:43.685127 19703 raft_consensus.cc:2229] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:19:43.685231 19703 raft_consensus.cc:2258] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:43.685238 19704 tablet_replica.cc:330] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: stopping tablet replica
I20260923 22:19:43.685318 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 19252
I20260923 22:19:43.685427 19704 raft_consensus.cc:2229] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:19:43.685562 19704 raft_consensus.cc:2258] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:19:43.685660 19703 ts_tablet_manager.cc:1905] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:43.687242 19704 ts_tablet_manager.cc:1905] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:19:43.688050 19703 ts_tablet_manager.cc:1918] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:43.688127 19703 log.cc:1196] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/18478aaecd5d46c684d1e6f8b6d48a3e
I20260923 22:19:43.688386 19703 ts_tablet_manager.cc:1939] T 18478aaecd5d46c684d1e6f8b6d48a3e P 9d149e4cc6c543798c8e6abc93f0ece3: Deleting consensus metadata
I20260923 22:19:43.688781 19201 catalog_manager.cc:4841] TS 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529): tablet 18478aaecd5d46c684d1e6f8b6d48a3e (table test-table [id=530aef9eb94e453aa3d69edb863e82cb]) successfully deleted
I20260923 22:19:43.689222 19704 ts_tablet_manager.cc:1918] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:19:43.689296 19704 log.cc:1196] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/18478aaecd5d46c684d1e6f8b6d48a3e
I20260923 22:19:43.689571 19704 ts_tablet_manager.cc:1939] T 18478aaecd5d46c684d1e6f8b6d48a3e P b9c595adacf24306b5742e7d15bfd6cf: Deleting consensus metadata
I20260923 22:19:43.689972 19198 catalog_manager.cc:4841] TS b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077): tablet 18478aaecd5d46c684d1e6f8b6d48a3e (table test-table [id=530aef9eb94e453aa3d69edb863e82cb]) successfully deleted
W20260923 22:19:43.692021 19198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:44725 (error 108)
I20260923 22:19:43.692302 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 19406
I20260923 22:19:43.693531 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:43.692520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:44725 (local address 127.0.0.1:45656)
0923 22:19:43.693449 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:43.693497 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:44725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":908}
W20260923 22:19:43.693678 19198 catalog_manager.cc:4584] TS a220a71fd3074789aabab378ad8233ce (127.4.207.1:44725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18478aaecd5d46c684d1e6f8b6d48a3e: Network error: Client connection negotiation failed: client connection to 127.4.207.1:44725: connect: Connection refused (error 111)
I20260923 22:19:43.700078 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 19544
I20260923 22:19:43.704882 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 19181
I20260923 22:19:43.709299 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42257
--webserver_interface=127.4.207.62
--webserver_port=38217
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:42257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:43.788538 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:43.788731 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:43.788753 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:43.789889 19706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:19:43.789932 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:43.789945 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:43.789958 19706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:19:43.789969 19706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:19:43.791484 19706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:42257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=38217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:43.791728 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:43.791962 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:43.796772 19711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:43.796861 19712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:43.796765 19714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:43.796949 19706 server_base.cc:1034] running on GCE node
I20260923 22:19:43.797204 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:43.797528 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:43.798695 19706 hybrid_clock.cc:648] HybridClock initialized: now 1790201983798682 us; error 43 us; skew 500 ppm
Sep 23 22:19:43 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:43.801261 19706 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:19:43.801723 19706 webserver.cc:460] Webserver started at http://127.4.207.62:38217/ using document root <none> and password file <none>
I20260923 22:19:43.801959 19706 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:43.802032 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:43.803323 19706 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:43.803999 19721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.804221 19706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.804306 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "88f233cbd5f44b8fb81bcebe3b5b388c"
format_stamp: "Formatted at 2026-09-23 22:19:41 on dist-test-slave-7511"
server_key: "80c6ae96b3dbb1deaa3f8f934bbe514c"
server_key_iv: "cfda1236979e65647546dee16c720c1e"
server_key_version: "encryptionkey@0"
I20260923 22:19:43.804621 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:43.822955 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:43.824848 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:43.824954 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:43.828907 19706 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:42257
I20260923 22:19:43.828977 19773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:42257 every 8 connection(s)
I20260923 22:19:43.829339 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:19:43.830896 19774 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:19:43.831590 19774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Bootstrap starting.
I20260923 22:19:43.833917 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 19706
I20260923 22:19:43.834168 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:44725
--local_ip_for_outbound_sockets=127.4.207.1
--tserver_master_addrs=127.4.207.62:42257
--webserver_port=34335
--webserver_interface=127.4.207.1
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:19:43.840185 19774 log.cc:824] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:43.842396 19774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:43.842687 19774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: Bootstrap complete.
I20260923 22:19:43.845129 19774 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:43.845378 19774 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f233cbd5f44b8fb81bcebe3b5b388c, State: Initialized, Role: FOLLOWER
I20260923 22:19:43.845578 19774 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:43.845667 19774 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:19:43.845721 19774 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:19:43.845772 19774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 2 FOLLOWER]: Advancing to term 3
I20260923 22:19:43.846750 19774 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:43.846868 19774 leader_election.cc:302] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f233cbd5f44b8fb81bcebe3b5b388c; no voters:
I20260923 22:19:43.847067 19774 leader_election.cc:288] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 22:19:43.847107 19778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 3 FOLLOWER]: Leader election won for term 3
I20260923 22:19:43.847236 19778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [term 3 LEADER]: Becoming Leader. State: Replica: 88f233cbd5f44b8fb81bcebe3b5b388c, State: Running, Role: LEADER
I20260923 22:19:43.847332 19774 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:19:43.847325 19778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } }
I20260923 22:19:43.847750 19779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f233cbd5f44b8fb81bcebe3b5b388c. Latest consensus state: current_term: 3 leader_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } } }
I20260923 22:19:43.847833 19779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:43.847755 19780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f233cbd5f44b8fb81bcebe3b5b388c" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42257 } } }
I20260923 22:19:43.847986 19780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c [sys.catalog]: This master's current role is: LEADER
I20260923 22:19:43.849359 19793 catalog_manager.cc:1259] Loaded cluster ID: 3db2b12ec87749c2a377e033d971a7f5
I20260923 22:19:43.849431 19793 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: loading cluster ID for follower catalog manager: success
W20260923 22:19:43.849655 19793 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 88f233cbd5f44b8fb81bcebe3b5b388c: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260923 22:19:43.849720 19793 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260923 22:19:43.849768 19783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:19:43.850087 19783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:19:43.850190 19783 catalog_manager.cc:1259] Loaded cluster ID: 3db2b12ec87749c2a377e033d971a7f5
I20260923 22:19:43.850222 19783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 22:19:43.850358 19783 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f70f77679ad google::LogMessage::Fail() at ??:0
@ 0x7f70f776a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f70f77674de google::LogMessage::Flush() at ??:0
@ 0x7f70f7768a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f70ff68c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f70ff6958b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f70f96350d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f70f962a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f70fa48e6db start_thread at ??:0
@ 0x7f70f626561f clone at ??:0
W20260923 22:19:43.942677 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:43.942881 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:43.942904 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:43.944116 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:43.944156 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:43.944181 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:19:43.945577 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:44725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=34335
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:43.945807 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:43.946062 19776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:43.952155 19811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:43.952253 19776 server_base.cc:1034] running on GCE node
W20260923 22:19:43.952193 19810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:43.952277 19813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:43.952514 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:43.952764 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:43.953924 19776 hybrid_clock.cc:648] HybridClock initialized: now 1790201983953911 us; error 30 us; skew 500 ppm
Sep 23 22:19:43 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:43.956444 19776 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:19:43.956889 19776 webserver.cc:460] Webserver started at http://127.4.207.1:34335/ using document root <none> and password file <none>
I20260923 22:19:43.957134 19776 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:43.957203 19776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:43.958683 19776 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:43.959488 19820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:43.959700 19776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:43.959796 19776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "a220a71fd3074789aabab378ad8233ce"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "6da5a608c0c9a4fe3ae82049ba736538"
server_key_iv: "bd73a497630ecab5d5b85f43ba944e61"
server_key_version: "encryptionkey@0"
I20260923 22:19:43.960251 19776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:43.986176 19776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:43.988968 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:43.989142 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:43.989414 19776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:43.989950 19827 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:19:43.991613 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:19:43.991678 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 22:19:43.991717 19776 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:19:43.992676 19776 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:19:43.992725 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:19:43.992839 19827 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Bootstrap starting.
I20260923 22:19:44.000679 19776 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:44725
I20260923 22:19:44.000764 19934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:44725 every 8 connection(s)
I20260923 22:19:44.001246 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:19:44.003222 19827 log.cc:824] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Log is configured to *not* fsync() on all Append() calls
I20260923 22:19:44.005303 19827 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:19:44.005585 19827 tablet_bootstrap.cc:492] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Bootstrap complete.
I20260923 22:19:44.006505 19827 ts_tablet_manager.cc:1397] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.009s
I20260923 22:19:44.007530 19827 raft_consensus.cc:348] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:44.007782 19827 raft_consensus.cc:729] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a220a71fd3074789aabab378ad8233ce, State: Initialized, Role: FOLLOWER
I20260923 22:19:44.007938 19827 consensus_queue.cc:260] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:44.008183 19827 ts_tablet_manager.cc:1428] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260923 22:19:44.009971 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 19776
I20260923 22:19:44.010242 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:33529
--local_ip_for_outbound_sockets=127.4.207.2
--tserver_master_addrs=127.4.207.62:42257
--webserver_port=46633
--webserver_interface=127.4.207.2
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:44.077248 19937 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:19:44.001960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:53153)
0923 22:19:44.002171 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.002222 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:44.002861 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:44.077016 (+ 74155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":93,"threads_started":1}
W20260923 22:19:44.077505 19935 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:42257 (0 consecutive failures): Network error: Failed to ping master at 127.4.207.62:42257: Client connection negotiation failed: client connection to 127.4.207.62:42257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 22:19:44.077908 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.077735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:39275)
0923 22:19:44.077801 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.077869 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 22:19:44.078290 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.078187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:37195)
0923 22:19:44.078236 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.078266 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 22:19:44.078433 19935 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:19:44.095213 19940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:44.095407 19940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:44.095429 19940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:44.096596 19940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:44.096653 19940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:44.096680 19940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:19:44.098016 19940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:33529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=46633
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:44.098204 19940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:44.098367 19940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:44.103164 19945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:44.103178 19946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:44.103178 19948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:44.103722 19940 server_base.cc:1034] running on GCE node
I20260923 22:19:44.103894 19940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:44.104099 19940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:44.105317 19940 hybrid_clock.cc:648] HybridClock initialized: now 1790201984105310 us; error 36 us; skew 500 ppm
Sep 23 22:19:44 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:44.108001 19940 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:19:44.108441 19940 webserver.cc:460] Webserver started at http://127.4.207.2:46633/ using document root <none> and password file <none>
I20260923 22:19:44.108706 19940 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:44.108778 19940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:44.110026 19940 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:19:44.110620 19955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:44.110770 19940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:44.110869 19940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "9d149e4cc6c543798c8e6abc93f0ece3"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "e326e9365f0f4311c64545722757e963"
server_key_iv: "3e981777585d53a659d9134665abc92c"
server_key_version: "encryptionkey@0"
I20260923 22:19:44.111187 19940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:44.129635 19940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:44.131618 19940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:44.131799 19940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:44.132018 19940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:44.132340 19940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:44.132421 19940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:44.132455 19940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:44.132479 19940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:44.137869 19940 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:33529
I20260923 22:19:44.137914 20068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:33529 every 8 connection(s)
I20260923 22:19:44.138244 19940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:19:44.139240 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.138886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:51737)
0923 22:19:44.139038 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.139106 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
W20260923 22:19:44.139731 20069 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:42257 (0 consecutive failures): Network error: Failed to ping master at 127.4.207.62:42257: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
I20260923 22:19:44.140019 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.139930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:33385)
0923 22:19:44.139971 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.140004 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 22:19:44.140306 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.140237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:36495)
0923 22:19:44.140270 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.140294 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260923 22:19:44.140441 20069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:19:44.145768 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 19940
I20260923 22:19:44.146023 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:42077
--local_ip_for_outbound_sockets=127.4.207.3
--tserver_master_addrs=127.4.207.62:42257
--webserver_port=34939
--webserver_interface=127.4.207.3
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:19:44.225306 20072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:19:44.225499 20072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:19:44.225520 20072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:19:44.226724 20072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:19:44.226775 20072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:19:44.226801 20072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:19:44.228796 20072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:42077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=34939
--tserver_master_addrs=127.4.207.62:42257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:19:44.229038 20072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:19:44.229267 20072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:19:44.234120 20080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:44.234128 20077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:19:44.234148 20078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:19:44.234783 20072 server_base.cc:1034] running on GCE node
I20260923 22:19:44.235015 20072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:19:44.235265 20072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:19:44.236428 20072 hybrid_clock.cc:648] HybridClock initialized: now 1790201984236396 us; error 44 us; skew 500 ppm
Sep 23 22:19:44 dist-test-slave-7511 krb5kdc[18636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201984, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:19:44.238933 20072 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:19:44.239285 20072 webserver.cc:460] Webserver started at http://127.4.207.3:34939/ using document root <none> and password file <none>
I20260923 22:19:44.239523 20072 fs_manager.cc:359] Metadata directory not provided
I20260923 22:19:44.239567 20072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:19:44.240733 20072 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:19:44.241271 20087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:44.241456 20072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:19:44.241523 20072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "b9c595adacf24306b5742e7d15bfd6cf"
format_stamp: "Formatted at 2026-09-23 22:19:42 on dist-test-slave-7511"
server_key: "38163d67227f18193b841ce7ef8ab32a"
server_key_iv: "dbb2145989a526714eb5177ae7170425"
server_key_version: "encryptionkey@0"
I20260923 22:19:44.241827 20072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:19:44.254333 20072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:19:44.256539 20072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:19:44.256647 20072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:19:44.256839 20072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:19:44.257136 20072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:19:44.257169 20072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:44.257190 20072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:19:44.257236 20072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:19:44.262814 20072 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:42077
I20260923 22:19:44.262888 20200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:42077 every 8 connection(s)
I20260923 22:19:44.263211 20072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:19:44.264317 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.263919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:39445)
0923 22:19:44.264111 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.264178 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
W20260923 22:19:44.264737 20201 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:42257 (0 consecutive failures): Network error: Failed to ping master at 127.4.207.62:42257: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
I20260923 22:19:44.265077 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.264929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:40571)
0923 22:19:44.264993 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.265027 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 22:19:44.265390 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:44.265300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:49903)
0923 22:19:44.265342 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:44.265369 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 22:19:44.265489 20201 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:19:44.270682 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 20072
I20260923 22:19:45.079107 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.078863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:42015)
0923 22:19:45.079004 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:45.079044 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260923 22:19:45.141000 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.140799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:37969)
0923 22:19:45.140916 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:45.140951 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260923 22:19:45.265959 20207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.265777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:38319)
0923 22:19:45.265880 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:45.265910 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260923 22:19:45.757097 20208 raft_consensus.cc:482] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:45.757216 20208 raft_consensus.cc:504] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:45.757596 20208 leader_election.cc:288] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529), b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077)
Sep 23 22:19:45 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.2@KRBTEST.COM
Sep 23 22:19:45 dist-test-slave-7511 krb5kdc[18636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790201983, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.3@KRBTEST.COM
I20260923 22:19:45.770411 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.757868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:57863 (local address 127.4.207.3:42077)
0923 22:19:45.758051 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:45.758057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:45.758085 (+ 28us) server_negotiation.cc:407] Connection header received
0923 22:19:45.758681 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:45.758684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:45.758745 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:45.758911 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:45.760011 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:45.761422 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:45.763077 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:45.763730 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:45.763887 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:45.767141 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:45.767145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:45.767145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:45.767156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:45.768802 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:45.769225 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:45.769227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:45.769228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:45.769305 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:45.769568 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:45.769569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:45.769569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:45.769769 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:45.769860 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:45.770170 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 22:19:45.770291 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
I20260923 22:19:45.770401 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.757751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:42077 (local address 127.4.207.1:57863)
0923 22:19:45.757969 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:45.757978 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:45.758034 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:45.759013 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:45.759014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:45.759593 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:45.759866 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:45.759874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:45.761552 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:45.761552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:45.762905 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:45.762908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:45.763870 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:45.763903 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:45.764026 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:45.764734 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:45.764738 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:45.767038 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:45.768892 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:45.768893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:45.768893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:45.769148 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:45.769400 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:45.769401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:45.769402 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:45.769469 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:45.769842 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:45.769848 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:45.770006 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 22:19:45.770107 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 22:19:45.770304 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":111,"thread_start_us":45,"threads_started":1}
I20260923 22:19:45.770529 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.757869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:44805 (local address 127.4.207.2:33529)
0923 22:19:45.758038 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 22:19:45.758051 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 22:19:45.758076 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:19:45.758681 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:19:45.758684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:19:45.758745 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:19:45.758911 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:19:45.760005 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:45.761422 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:45.763056 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:19:45.763730 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:19:45.763855 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:45.766764 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:19:45.766768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:19:45.766769 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:19:45.766779 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:19:45.768673 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:45.769179 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:45.769182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:45.769183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:45.769285 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:19:45.769528 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:19:45.769529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:19:45.769529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:19:45.769768 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:19:45.769860 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 22:19:45.770290 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 22:19:45.770430 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260923 22:19:45.770526 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:45.757751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:33529 (local address 127.4.207.1:44805)
0923 22:19:45.757856 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:45.757867 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:19:45.757943 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:19:45.758986 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:19:45.758990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:19:45.759593 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:19:45.759866 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:45.759874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:45.761547 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:45.761549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:45.762905 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:19:45.762908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:19:45.763836 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:19:45.763837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:19:45.764027 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:19:45.764724 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:19:45.764734 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:19:45.766641 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:19:45.768762 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:45.768768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:45.768770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:45.769102 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:45.769351 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:19:45.769351 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:19:45.769352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:19:45.769463 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:19:45.769841 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:19:45.769847 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:19:45.770023 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 22:19:45.770102 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 22:19:45.770394 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":199,"thread_start_us":56,"threads_started":1}
W20260923 22:19:45.771363 19824 leader_election.cc:341] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529): Not found: Tablet not found: 18478aaecd5d46c684d1e6f8b6d48a3e
W20260923 22:19:45.771847 19821 leader_election.cc:341] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077): Not found: Tablet not found: 18478aaecd5d46c684d1e6f8b6d48a3e
I20260923 22:19:45.771914 19821 leader_election.cc:302] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a220a71fd3074789aabab378ad8233ce; no voters: 9d149e4cc6c543798c8e6abc93f0ece3, b9c595adacf24306b5742e7d15bfd6cf
I20260923 22:19:45.772023 20208 raft_consensus.cc:2726] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:46.079859 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:46.079652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:49859)
0923 22:19:46.079696 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:46.079805 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 22:19:46.141781 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:46.141531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:41739)
0923 22:19:46.141680 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:46.141734 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260923 22:19:46.266602 20214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:46.266390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:50053)
0923 22:19:46.266505 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:46.266558 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260923 22:19:47.080484 20215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:47.080282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:37657)
0923 22:19:47.080404 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:47.080438 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260923 22:19:47.142489 20216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:47.142314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:48911)
0923 22:19:47.142421 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:47.142449 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260923 22:19:47.267269 20217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:47.267040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:54171)
0923 22:19:47.267175 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:47.267220 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260923 22:19:47.836323 20218 raft_consensus.cc:482] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:47.836431 20218 raft_consensus.cc:504] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:47.836649 20218 leader_election.cc:288] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529), b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077)
W20260923 22:19:47.837070 19821 leader_election.cc:341] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077): Not found: Tablet not found: 18478aaecd5d46c684d1e6f8b6d48a3e
W20260923 22:19:47.837153 19824 leader_election.cc:341] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529): Not found: Tablet not found: 18478aaecd5d46c684d1e6f8b6d48a3e
I20260923 22:19:47.837260 19824 leader_election.cc:302] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a220a71fd3074789aabab378ad8233ce; no voters: 9d149e4cc6c543798c8e6abc93f0ece3, b9c595adacf24306b5742e7d15bfd6cf
I20260923 22:19:47.837350 20218 raft_consensus.cc:2726] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:48.081203 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:48.080967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:42483)
0923 22:19:48.081102 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:48.081141 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260923 22:19:48.143124 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:48.142912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:36561)
0923 22:19:48.143028 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:48.143070 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260923 22:19:48.267962 20221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:48.267757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:46507)
0923 22:19:48.267872 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:48.267908 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260923 22:19:49.081916 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:49.081704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:37913)
0923 22:19:49.081831 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:49.081869 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260923 22:19:49.143908 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:49.143700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:53623)
0923 22:19:49.143826 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:49.143867 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260923 22:19:49.268759 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:49.268529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:53365)
0923 22:19:49.268678 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:49.268716 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260923 22:19:50.082599 20225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:50.082353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:32813)
0923 22:19:50.082492 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:50.082552 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260923 22:19:50.120620 20226 raft_consensus.cc:482] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:50.120710 20226 raft_consensus.cc:504] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:50.120862 20226 leader_election.cc:288] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529), b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077)
W20260923 22:19:50.121268 19821 leader_election.cc:341] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077): Not found: Tablet not found: 18478aaecd5d46c684d1e6f8b6d48a3e
W20260923 22:19:50.121366 19824 leader_election.cc:341] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529): Not found: Tablet not found: 18478aaecd5d46c684d1e6f8b6d48a3e
I20260923 22:19:50.121451 19824 leader_election.cc:302] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a220a71fd3074789aabab378ad8233ce; no voters: 9d149e4cc6c543798c8e6abc93f0ece3, b9c595adacf24306b5742e7d15bfd6cf
I20260923 22:19:50.121580 20226 raft_consensus.cc:2726] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:50.144614 20227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:50.144381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:33037)
0923 22:19:50.144505 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:50.144564 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260923 22:19:50.269526 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:50.269313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:48037)
0923 22:19:50.269449 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:50.269483 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260923 22:19:51.083207 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:51.083009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:40011)
0923 22:19:51.083130 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:51.083163 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260923 22:19:51.145231 20230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:51.145038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:43375)
0923 22:19:51.145157 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:51.145189 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260923 22:19:51.270260 20231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:51.270026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:53691)
0923 22:19:51.270179 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:51.270215 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":61,"threads_started":1}
I20260923 22:19:52.083948 20232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:52.083705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:60149)
0923 22:19:52.083859 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:52.083901 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260923 22:19:52.145968 20233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:52.145743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:38223)
0923 22:19:52.145874 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:52.145918 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260923 22:19:52.271080 20234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:52.270792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:35051)
0923 22:19:52.270989 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:52.271032 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260923 22:19:53.084628 20235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:53.084385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:56535)
0923 22:19:53.084540 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:53.084583 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260923 22:19:53.146658 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:53.146445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:50005)
0923 22:19:53.146574 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:53.146611 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260923 22:19:53.271850 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:53.271623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:52783)
0923 22:19:53.271771 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:53.271807 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260923 22:19:53.857200 20238 raft_consensus.cc:482] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:19:53.857335 20238 raft_consensus.cc:504] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d149e4cc6c543798c8e6abc93f0ece3" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 33529 } } peers { permanent_uuid: "b9c595adacf24306b5742e7d15bfd6cf" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42077 } } peers { permanent_uuid: "a220a71fd3074789aabab378ad8233ce" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44725 } }
I20260923 22:19:53.857514 20238 leader_election.cc:288] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529), b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077)
W20260923 22:19:53.857882 19821 leader_election.cc:341] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9c595adacf24306b5742e7d15bfd6cf (127.4.207.3:42077): Not found: Tablet not found: 18478aaecd5d46c684d1e6f8b6d48a3e
W20260923 22:19:53.857957 19824 leader_election.cc:341] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9d149e4cc6c543798c8e6abc93f0ece3 (127.4.207.2:33529): Not found: Tablet not found: 18478aaecd5d46c684d1e6f8b6d48a3e
I20260923 22:19:53.858047 19824 leader_election.cc:302] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a220a71fd3074789aabab378ad8233ce; no voters: 9d149e4cc6c543798c8e6abc93f0ece3, b9c595adacf24306b5742e7d15bfd6cf
I20260923 22:19:53.858119 20238 raft_consensus.cc:2726] T 18478aaecd5d46c684d1e6f8b6d48a3e P a220a71fd3074789aabab378ad8233ce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:19:54.085557 20239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:54.085276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:33399)
0923 22:19:54.085414 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:54.085460 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
I20260923 22:19:54.147311 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:54.147094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:35769)
0923 22:19:54.147202 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:54.147237 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260923 22:19:54.272540 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:54.272326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:43677)
0923 22:19:54.272456 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:54.272495 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260923 22:19:55.086340 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:55.086079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:41143)
0923 22:19:55.086254 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:55.086292 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":81,"threads_started":1}
I20260923 22:19:55.147909 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:55.147688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:46423)
0923 22:19:55.147824 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:55.147862 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260923 22:19:55.273169 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:55.272972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:37153)
0923 22:19:55.273092 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:55.273125 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260923 22:19:56.086969 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.086762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:43277)
0923 22:19:56.086884 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:56.086922 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260923 22:19:56.148530 20246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.148338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:55445)
0923 22:19:56.148451 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:56.148485 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260923 22:19:56.273885 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:56.273660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:58971)
0923 22:19:56.273800 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:56.273839 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260923 22:19:57.087875 20248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.087586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:45129)
0923 22:19:57.087783 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.087826 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
I20260923 22:19:57.149214 20249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.148988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:41611)
0923 22:19:57.149126 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.149166 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260923 22:19:57.274663 20250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:57.274419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:49699)
0923 22:19:57.274576 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:57.274617 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260923 22:19:58.088611 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:58.088406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:59707)
0923 22:19:58.088528 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:58.088566 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260923 22:19:58.149864 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:58.149657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:45553)
0923 22:19:58.149785 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:58.149817 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":56,"threads_started":1}
I20260923 22:19:58.275481 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:58.275186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:57601)
0923 22:19:58.275367 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:58.275400 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":118,"threads_started":1}
I20260923 22:19:59.089329 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:59.089125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.1:40939)
0923 22:19:59.089245 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:59.089283 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260923 22:19:59.150588 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:59.150351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.2:36785)
0923 22:19:59.150483 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:59.150520 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260923 22:19:59.276177 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:19:59.275960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42257 (local address 127.4.207.3:47465)
0923 22:19:59.276080 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 22:19:59.276132 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:42257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260923 22:19:59.280869 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 19776
I20260923 22:19:59.285619 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 19940
I20260923 22:19:59.289849 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 20072
2026-09-23T22:19:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18543 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:19:59 dist-test-slave-7511 krb5kdc[20259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:19:59 dist-test-slave-7511 krb5kdc[20259](info): set up 2 sockets
Sep 23 22:19:59 dist-test-slave-7511 krb5kdc[20259](info): commencing operation
krb5kdc: starting...
W20260923 22:20:00.311628 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:20:00 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:20:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:20:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:20:00.429849 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:38315
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:38315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:00.510737 20272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:00.510972 20272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:00.510993 20272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:00.512116 20272 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 22:20:00.512152 20272 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 22:20:00.512166 20272 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 22:20:00.512178 20272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:00.512190 20272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:00.512203 20272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:00.512215 20272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:00.512230 20272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:00.513621 20272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:38315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:38315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:00.513806 20272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:00.513968 20272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:00.519362 20278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:00.519354 20277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:00.519484 20272 server_base.cc:1034] running on GCE node
W20260923 22:20:00.519354 20280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:00.519889 20272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:00.520200 20272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:00.521363 20272 hybrid_clock.cc:648] HybridClock initialized: now 1790202000521350 us; error 45 us; skew 500 ppm
Sep 23 22:20:00 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202000, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:00.523950 20272 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:20:00.524430 20272 webserver.cc:460] Webserver started at http://127.4.207.62:46781/ using document root <none> and password file <none>
I20260923 22:20:00.524682 20272 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:00.524751 20272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:00.524892 20272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:00.525926 20272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "67e597cddb5a4d60a997b1d016783c43"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "80104879d318f9ccc084ed593a8ac22e"
server_key_iv: "c6a669fc66675e1658eba12034035c98"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.526285 20272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "67e597cddb5a4d60a997b1d016783c43"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "80104879d318f9ccc084ed593a8ac22e"
server_key_iv: "c6a669fc66675e1658eba12034035c98"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.527698 20272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:20:00.528462 20287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:00.528648 20272 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:00.528759 20272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "67e597cddb5a4d60a997b1d016783c43"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "80104879d318f9ccc084ed593a8ac22e"
server_key_iv: "c6a669fc66675e1658eba12034035c98"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.528836 20272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:00.555821 20272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:00.558251 20272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:00.558456 20272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:00.562618 20272 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:38315
I20260923 22:20:00.562650 20339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:38315 every 8 connection(s)
I20260923 22:20:00.563004 20272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:00.563899 20340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:00.564962 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 20272
I20260923 22:20:00.565065 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:20:00.565303 4924 external_mini_cluster.cc:1428] Setting key aa3a6253f932d3e6eaaec77310a0e804
I20260923 22:20:00.566699 20340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Bootstrap starting.
I20260923 22:20:00.567525 20340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:00.567852 20340 log.cc:824] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:00.568564 20340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: No bootstrap required, opened a new log
I20260923 22:20:00.570991 20340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:00.571170 20340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:00.571209 20340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e597cddb5a4d60a997b1d016783c43, State: Initialized, Role: FOLLOWER
I20260923 22:20:00.571364 20340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:00.571486 20340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:00.571533 20340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:00.571590 20340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:00.572299 20340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:00.572389 20340 leader_election.cc:302] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67e597cddb5a4d60a997b1d016783c43; no voters:
I20260923 22:20:00.572585 20340 leader_election.cc:288] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:20:00.572613 20345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 1 FOLLOWER]: Leader election won for term 1
Sep 23 22:20:00 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:00.572727 20345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 1 LEADER]: Becoming Leader. State: Replica: 67e597cddb5a4d60a997b1d016783c43, State: Running, Role: LEADER
I20260923 22:20:00.572886 20340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:00.572880 20345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:00.573345 20345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67e597cddb5a4d60a997b1d016783c43. Latest consensus state: current_term: 1 leader_uuid: "67e597cddb5a4d60a997b1d016783c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } } }
I20260923 22:20:00.573583 20345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:00.573503 20347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67e597cddb5a4d60a997b1d016783c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } } }
I20260923 22:20:00.573727 20347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:00.573887 20352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:00.574604 20352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:00.576167 20352 catalog_manager.cc:1347] Generated new cluster ID: a63892d414ff43768252e0dea078e323
I20260923 22:20:00.576216 20352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:20:00.577206 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:00.565881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.4.207.62:38315)
0923 22:20:00.566073 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:00.566081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:00.566107 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:20:00.566513 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:00.566519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:00.566565 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:00.566690 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:00.567540 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.568773 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.569832 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.570267 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.570350 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:00.573225 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:00.573229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:00.573229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:00.573240 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:00.575153 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:00.575605 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:00.575608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:00.575608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:00.575711 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:00.575964 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:00.575965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:00.575966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:00.576280 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:00.576432 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:00.576869 (+ 437us) server_negotiation.cc:299] Negotiation successful
0923 22:20:00.576989 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260923 22:20:00.585057 20352 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:20:00.585583 20352 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:00.591396 20352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Generated new TSK 0
I20260923 22:20:00.591601 20352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:20:00.622187 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38315
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:00.704319 20366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:00.704552 20366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:00.704600 20366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:00.705732 20366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:00.705803 20366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:00.705847 20366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:20:00.707284 20366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:00.707595 20366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:00.707821 20366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:00.712863 20372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:00.712873 20374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:00.712890 20366 server_base.cc:1034] running on GCE node
W20260923 22:20:00.712880 20371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:00.713335 20366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:00.713575 20366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:00.714758 20366 hybrid_clock.cc:648] HybridClock initialized: now 1790202000714741 us; error 30 us; skew 500 ppm
Sep 23 22:20:00 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202000, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:00.717463 20366 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:20:00.717922 20366 webserver.cc:460] Webserver started at http://127.4.207.1:34973/ using document root <none> and password file <none>
I20260923 22:20:00.718163 20366 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:00.718235 20366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:00.718402 20366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:00.719602 20366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "7b6754039a6d4657bbca17891f115631"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "ec99ecc1f43fb73b4edebc4543915d96"
server_key_iv: "352cfbba9c0d58d8286b9006228decad"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.719992 20366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "7b6754039a6d4657bbca17891f115631"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "ec99ecc1f43fb73b4edebc4543915d96"
server_key_iv: "352cfbba9c0d58d8286b9006228decad"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.721349 20366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:20:00.722111 20381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:00.722287 20366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:00.722386 20366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "7b6754039a6d4657bbca17891f115631"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "ec99ecc1f43fb73b4edebc4543915d96"
server_key_iv: "352cfbba9c0d58d8286b9006228decad"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.722471 20366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:00.743137 20366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:00.745105 20366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:00.745268 20366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:00.745510 20366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:00.745836 20366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:00.745893 20366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:00.745936 20366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:00.745959 20366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:00.751637 20366 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:45671
I20260923 22:20:00.751688 20494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:45671 every 8 connection(s)
I20260923 22:20:00.752064 20366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:00.757758 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 20366
I20260923 22:20:00.757869 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:20:00.758122 4924 external_mini_cluster.cc:1428] Setting key c6b3c6ebde159d1164f4966f69bb77bc
Sep 23 22:20:00 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202000, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:00.762501 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:00.752870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:33779 (local address 127.4.207.62:38315)
0923 22:20:00.752959 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:00.752961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:00.753170 (+ 209us) server_negotiation.cc:407] Connection header received
0923 22:20:00.753737 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:00.753738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:00.753751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:00.753816 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:00.754822 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.755359 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.756549 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.756769 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.756875 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:00.759378 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:00.759379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:00.759380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:00.759388 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:00.760942 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:00.761468 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:00.761470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:00.761470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:00.761538 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:00.761800 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:00.761801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:00.761802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:00.761926 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:00.762005 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:00.762306 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 22:20:00.762409 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:20:00.762698 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:00.752853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:33779)
0923 22:20:00.753027 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:00.753063 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:00.753608 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:00.753897 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:00.753901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:00.754482 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:00.754699 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:00.754703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.755493 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.755495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:00.756458 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:00.756460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.756885 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.756886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:00.757011 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:00.757562 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:00.757572 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:00.759282 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:00.761028 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:00.761036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:00.761039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:00.761398 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:00.761609 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:00.761610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:00.761610 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:00.761730 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:00.761996 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:00.762003 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:00.762164 (+ 161us) client_negotiation.cc:769] Sending connection context
0923 22:20:00.762272 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 22:20:00.762442 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260923 22:20:00.763228 20495 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38315
I20260923 22:20:00.763326 20495 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:00.763561 20495 heartbeater.cc:511] Master 127.4.207.62:38315 requested a full tablet report, sending...
I20260923 22:20:00.764187 20304 ts_manager.cc:194] Registered new tserver with Master: 7b6754039a6d4657bbca17891f115631 (127.4.207.1:45671)
I20260923 22:20:00.765061 20304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:33779
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:20:00.784654 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38315
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:00.867266 20500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:00.867501 20500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:00.867525 20500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:00.868713 20500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:00.868759 20500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:00.868788 20500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:20:00.870308 20500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:00.870548 20500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:00.870779 20500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:00.876132 20508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:00.876133 20505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:00.876188 20506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:00.876190 20500 server_base.cc:1034] running on GCE node
I20260923 22:20:00.876544 20500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:00.876823 20500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:00.877962 20500 hybrid_clock.cc:648] HybridClock initialized: now 1790202000877942 us; error 35 us; skew 500 ppm
Sep 23 22:20:00 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202000, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:00.880411 20500 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:20:00.880882 20500 webserver.cc:460] Webserver started at http://127.4.207.2:34881/ using document root <none> and password file <none>
I20260923 22:20:00.881101 20500 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:00.881168 20500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:00.881309 20500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:00.882359 20500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "42310e349a584898ae2f3aa4ea51f4c9"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "0389cf18a71353c203219f6c9cd6c621"
server_key_iv: "5301ad36f643e7352c6506137875d657"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.882726 20500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "42310e349a584898ae2f3aa4ea51f4c9"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "0389cf18a71353c203219f6c9cd6c621"
server_key_iv: "5301ad36f643e7352c6506137875d657"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.884150 20500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:20:00.884867 20515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:00.885046 20500 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:00.885145 20500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "42310e349a584898ae2f3aa4ea51f4c9"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "0389cf18a71353c203219f6c9cd6c621"
server_key_iv: "5301ad36f643e7352c6506137875d657"
server_key_version: "encryptionkey@0"
I20260923 22:20:00.885222 20500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:00.907433 20500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:00.909660 20500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:00.909813 20500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:00.910043 20500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:00.910374 20500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:00.910425 20500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:00.910465 20500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:00.910490 20500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:00.916508 20500 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:39037
I20260923 22:20:00.916554 20628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:39037 every 8 connection(s)
I20260923 22:20:00.916867 20500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:20:00.919749 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 20500
I20260923 22:20:00.919834 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:20:00.920045 4924 external_mini_cluster.cc:1428] Setting key 29a3e5328d3979e8290bb546b6fcec0b
Sep 23 22:20:00 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202000, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:00.927219 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:00.917542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:52567 (local address 127.4.207.62:38315)
0923 22:20:00.917624 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:00.917626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:00.917857 (+ 231us) server_negotiation.cc:407] Connection header received
0923 22:20:00.918348 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:00.918349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:00.918359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:00.918415 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:00.919349 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.919961 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.921350 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.921532 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.921580 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:00.924147 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:00.924148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:00.924148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:00.924156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:00.925750 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:00.926159 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:00.926160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:00.926161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:00.926217 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:00.926532 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:00.926533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:00.926534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:00.926689 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:00.926767 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:00.927007 (+ 240us) server_negotiation.cc:299] Negotiation successful
0923 22:20:00.927105 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 22:20:00.927242 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:00.917539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:52567)
0923 22:20:00.917717 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:00.917759 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:00.918264 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:00.918470 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:00.918473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:00.919064 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:00.919266 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:00.919271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.920047 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.920048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:00.921259 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:00.921262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:00.921609 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:00.921610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:00.921754 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:00.922383 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:00.922393 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:00.924013 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:00.925832 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:00.925839 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:00.925841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:00.926065 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:00.926323 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:00.926323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:00.926324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:00.926459 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:00.926761 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:00.926766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:00.926876 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 22:20:00.926960 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:20:00.927082 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":109,"threads_started":1}
I20260923 22:20:00.927839 20629 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38315
I20260923 22:20:00.927940 20629 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:00.928107 20629 heartbeater.cc:511] Master 127.4.207.62:38315 requested a full tablet report, sending...
I20260923 22:20:00.928540 20304 ts_manager.cc:194] Registered new tserver with Master: 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037)
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
I20260923 22:20:00.929039 20304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:52567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:20:00.946185 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38315
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:01.026598 20634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:01.026803 20634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:01.026827 20634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:01.027941 20634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:01.027979 20634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:01.028005 20634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:20:01.029381 20634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:01.029561 20634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:01.029748 20634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:01.035008 20642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:01.034987 20639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:01.035004 20634 server_base.cc:1034] running on GCE node
W20260923 22:20:01.034971 20640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:01.035526 20634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:01.035778 20634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:01.036937 20634 hybrid_clock.cc:648] HybridClock initialized: now 1790202001036926 us; error 21 us; skew 500 ppm
Sep 23 22:20:01 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202001, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:01.039281 20634 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:20:01.039745 20634 webserver.cc:460] Webserver started at http://127.4.207.3:42551/ using document root <none> and password file <none>
I20260923 22:20:01.039924 20634 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:01.039963 20634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:01.040048 20634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:01.041075 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "fd7d617f91a3413988462c7963e04116"
format_stamp: "Formatted at 2026-09-23 22:20:01 on dist-test-slave-7511"
server_key: "c458d45aadf41bb74e168f66c8715b1c"
server_key_iv: "b680c30590dffde97f7c16b4bddd694b"
server_key_version: "encryptionkey@0"
I20260923 22:20:01.041378 20634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "fd7d617f91a3413988462c7963e04116"
format_stamp: "Formatted at 2026-09-23 22:20:01 on dist-test-slave-7511"
server_key: "c458d45aadf41bb74e168f66c8715b1c"
server_key_iv: "b680c30590dffde97f7c16b4bddd694b"
server_key_version: "encryptionkey@0"
I20260923 22:20:01.042626 20634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:20:01.043298 20649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:01.043497 20634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:20:01.043583 20634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "fd7d617f91a3413988462c7963e04116"
format_stamp: "Formatted at 2026-09-23 22:20:01 on dist-test-slave-7511"
server_key: "c458d45aadf41bb74e168f66c8715b1c"
server_key_iv: "b680c30590dffde97f7c16b4bddd694b"
server_key_version: "encryptionkey@0"
I20260923 22:20:01.043638 20634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:01.056591 20634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:01.058770 20634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:01.058879 20634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:01.059039 20634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:01.059301 20634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:01.059332 20634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:01.059409 20634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:01.059453 20634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:01.065271 20634 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:33009
I20260923 22:20:01.065358 20762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:33009 every 8 connection(s)
I20260923 22:20:01.065673 20634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:20:01.071012 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 20634
I20260923 22:20:01.071142 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:20:01.071375 4924 external_mini_cluster.cc:1428] Setting key ee72fe7087de319d643ca54ce25b7136
Sep 23 22:20:01 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202001, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:01.075326 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.066478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:47139 (local address 127.4.207.62:38315)
0923 22:20:01.066567 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:01.066570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:01.066755 (+ 185us) server_negotiation.cc:407] Connection header received
0923 22:20:01.067281 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:01.067282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:01.067294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:01.067360 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:01.068423 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.068941 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.069932 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.070147 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.070233 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.072691 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:01.072692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:01.072693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:01.072700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:01.074057 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:01.074408 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:01.074409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:01.074409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:01.074465 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:01.074719 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:01.074720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:01.074720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:01.074864 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:01.074918 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:01.075200 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 22:20:01.075267 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 22:20:01.075507 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.066441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:47139)
0923 22:20:01.066620 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:01.066668 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:01.067177 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:01.067480 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:01.067486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:01.068103 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:01.068307 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.068324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.069036 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.069037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.069841 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.069843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.070242 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.070243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.070370 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.070914 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:01.070922 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:01.072586 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:01.074126 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:01.074133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:01.074134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:01.074342 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:01.074522 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:01.074523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:01.074523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:01.074662 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:01.074929 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:01.074935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:01.075086 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 22:20:01.075177 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 22:20:01.075293 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260923 22:20:01.075976 20763 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38315
I20260923 22:20:01.076076 20763 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:01.076248 20763 heartbeater.cc:511] Master 127.4.207.62:38315 requested a full tablet report, sending...
I20260923 22:20:01.076623 20304 ts_manager.cc:194] Registered new tserver with Master: fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009)
I20260923 22:20:01.076982 20304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:47139
I20260923 22:20:01.083297 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:20:01.095384 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.084569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.4.207.62:38315)
0923 22:20:01.084621 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:01.084623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:01.084709 (+ 86us) server_negotiation.cc:407] Connection header received
0923 22:20:01.084747 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:01.084747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:01.084756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:01.084808 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:01.085312 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.085762 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.086445 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.086602 (+ 157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.086646 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.092971 (+ 6325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:01.092973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:01.092973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:01.092980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:01.094404 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:01.094729 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:01.094730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:01.094730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:01.094779 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:01.094935 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:01.094936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:01.094936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:01.095073 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:01.095149 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:01.095238 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 22:20:01.095294 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:20:01.097169 20304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:20:01.098752 20304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:20:01.105122 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.100975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:45671 (local address 127.0.0.1:46062)
0923 22:20:01.101231 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:01.101239 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:01.101278 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:01.101575 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:01.101578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:01.101580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:01.101647 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.101649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.103006 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.103006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.104187 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.104189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.104837 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.104907 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.104920 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 22:20:01.104980 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 22:20:01.105040 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":33,"thread_start_us":41,"threads_started":1}
I20260923 22:20:01.105208 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.101065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.4.207.2:39037)
0923 22:20:01.101703 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:01.101707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:01.101716 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:20:01.101754 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:01.101756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:01.101770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:01.101840 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:01.102051 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.102763 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104130 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.104812 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104884 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.104943 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:01.105071 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 22:20:01.105124 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":69,"threads_started":1}
I20260923 22:20:01.105321 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.101225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.4.207.1:45671)
0923 22:20:01.101406 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:01.101409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:01.101418 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:20:01.101457 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:01.101459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:01.101475 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:01.101563 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:01.101738 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.102901 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104289 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.104741 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104833 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.104893 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:01.105182 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 22:20:01.105238 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
I20260923 22:20:01.105396 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.101198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.4.207.3:33009)
0923 22:20:01.101380 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:01.101386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:01.101397 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:20:01.101457 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:01.101459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:01.101475 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:01.101563 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:01.101782 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.102913 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104259 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.104861 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104914 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.104956 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:01.105277 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 22:20:01.105323 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260923 22:20:01.105221 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.100989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33009 (local address 127.0.0.1:47784)
0923 22:20:01.101356 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:01.101364 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:01.101394 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:01.101575 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:01.101578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:01.101582 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:01.101700 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.101702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.103110 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.103111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.104181 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.104184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.104987 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.105068 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.105073 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:20:01.105120 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 22:20:01.105158 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":71,"thread_start_us":79,"threads_started":1}
I20260923 22:20:01.105127 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.100974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:39037 (local address 127.0.0.1:40378)
0923 22:20:01.101148 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:01.101189 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:01.101262 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:01.101922 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:01.101923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:01.101925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:01.101967 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.101968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.102867 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.102868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.104051 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.104053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.104936 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.104937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.104992 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.104995 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:01.105043 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:20:01.105072 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":80,"thread_start_us":111,"threads_started":1}
I20260923 22:20:01.106230 20563 tablet_service.cc:1467] Processing CreateTablet for tablet d2c60f297c7c40afaf9e2b86b7dda998 (DEFAULT_TABLE table=test-table [id=62f7a1e91649465cacac4e0ca172269c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:01.106227 20697 tablet_service.cc:1467] Processing CreateTablet for tablet d2c60f297c7c40afaf9e2b86b7dda998 (DEFAULT_TABLE table=test-table [id=62f7a1e91649465cacac4e0ca172269c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:01.106227 20429 tablet_service.cc:1467] Processing CreateTablet for tablet d2c60f297c7c40afaf9e2b86b7dda998 (DEFAULT_TABLE table=test-table [id=62f7a1e91649465cacac4e0ca172269c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:01.106580 20563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c60f297c7c40afaf9e2b86b7dda998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:01.106580 20697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c60f297c7c40afaf9e2b86b7dda998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:01.106580 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2c60f297c7c40afaf9e2b86b7dda998. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:01.109351 20780 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Bootstrap starting.
I20260923 22:20:01.109766 20782 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Bootstrap starting.
I20260923 22:20:01.109961 20781 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: Bootstrap starting.
I20260923 22:20:01.110198 20780 tablet_bootstrap.cc:654] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:01.110430 20782 tablet_bootstrap.cc:654] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:01.110638 20781 tablet_bootstrap.cc:654] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:01.110849 20780 log.cc:824] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:01.111034 20782 log.cc:824] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:01.111198 20781 log.cc:824] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:01.111672 20780 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: No bootstrap required, opened a new log
I20260923 22:20:01.111763 20780 ts_tablet_manager.cc:1397] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 22:20:01.111966 20781 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: No bootstrap required, opened a new log
I20260923 22:20:01.112057 20781 ts_tablet_manager.cc:1397] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:20:01.113210 20780 raft_consensus.cc:348] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.113387 20780 raft_consensus.cc:374] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:01.113315 20781 raft_consensus.cc:348] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.113432 20780 raft_consensus.cc:729] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42310e349a584898ae2f3aa4ea51f4c9, State: Initialized, Role: FOLLOWER
I20260923 22:20:01.113476 20781 raft_consensus.cc:374] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:01.113514 20781 raft_consensus.cc:729] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7d617f91a3413988462c7963e04116, State: Initialized, Role: FOLLOWER
I20260923 22:20:01.113546 20780 consensus_queue.cc:260] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.113618 20781 consensus_queue.cc:260] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.113803 20780 ts_tablet_manager.cc:1428] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:01.113900 20781 ts_tablet_manager.cc:1428] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:20:01.113963 20763 heartbeater.cc:503] Master 127.4.207.62:38315 was elected leader, sending a full tablet report...
I20260923 22:20:01.114003 20629 heartbeater.cc:503] Master 127.4.207.62:38315 was elected leader, sending a full tablet report...
I20260923 22:20:01.114347 20782 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: No bootstrap required, opened a new log
I20260923 22:20:01.114410 20782 ts_tablet_manager.cc:1397] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260923 22:20:01.116132 20782 raft_consensus.cc:348] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.116272 20782 raft_consensus.cc:374] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:01.116313 20782 raft_consensus.cc:729] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6754039a6d4657bbca17891f115631, State: Initialized, Role: FOLLOWER
I20260923 22:20:01.116413 20782 consensus_queue.cc:260] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.116654 20495 heartbeater.cc:503] Master 127.4.207.62:38315 was elected leader, sending a full tablet report...
I20260923 22:20:01.116655 20782 ts_tablet_manager.cc:1428] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260923 22:20:01.167486 20630 tablet.cc:2369] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:01.207858 20788 raft_consensus.cc:482] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:01.207998 20788 raft_consensus.cc:504] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.208455 20788 leader_election.cc:288] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037), fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009)
I20260923 22:20:01.211355 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.208613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:39037 (local address 127.4.207.1:51965)
0923 22:20:01.208680 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:01.208694 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:01.208746 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:01.208984 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:01.208986 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:01.208988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:01.209068 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.209070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.209723 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.209724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.210458 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.210459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.211045 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.211047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.211138 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.211141 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:01.211201 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 22:20:01.211254 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 22:20:01.211383 20778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.208750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:35585 (local address 127.4.207.3:33009)
0923 22:20:01.208811 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:01.208813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:01.208959 (+ 146us) server_negotiation.cc:407] Connection header received
0923 22:20:01.208988 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:01.208989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:01.209001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:01.209088 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:01.209241 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.209664 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.210617 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.210988 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.211096 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.211106 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:01.211265 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 22:20:01.211319 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:20:01.211354 20777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.208742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:51965 (local address 127.4.207.2:39037)
0923 22:20:01.208824 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:01.208827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:01.208836 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:20:01.208867 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:01.208868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:01.208881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:01.208938 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:01.209158 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.209641 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.210543 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.210967 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.211017 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.211024 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:01.211223 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 22:20:01.211265 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 22:20:01.211661 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.208642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33009 (local address 127.4.207.1:35585)
0923 22:20:01.208852 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:01.208862 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:01.208920 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:01.209083 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:01.209083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:01.209085 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:01.209173 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.209175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.209787 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.209787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.210520 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:01.210523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.211112 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.211112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:01.211166 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.211181 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 22:20:01.211250 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 22:20:01.211303 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":126,"threads_started":1}
I20260923 22:20:01.211763 20583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c60f297c7c40afaf9e2b86b7dda998" candidate_uuid: "7b6754039a6d4657bbca17891f115631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42310e349a584898ae2f3aa4ea51f4c9" is_pre_election: true
I20260923 22:20:01.211946 20583 raft_consensus.cc:2454] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6754039a6d4657bbca17891f115631 in term 0.
I20260923 22:20:01.212036 20717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c60f297c7c40afaf9e2b86b7dda998" candidate_uuid: "7b6754039a6d4657bbca17891f115631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7d617f91a3413988462c7963e04116" is_pre_election: true
I20260923 22:20:01.212206 20717 raft_consensus.cc:2454] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6754039a6d4657bbca17891f115631 in term 0.
I20260923 22:20:01.212188 20385 leader_election.cc:302] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42310e349a584898ae2f3aa4ea51f4c9, 7b6754039a6d4657bbca17891f115631; no voters:
I20260923 22:20:01.212327 20788 raft_consensus.cc:2781] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:01.212407 20788 raft_consensus.cc:482] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:01.212441 20788 raft_consensus.cc:3037] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:01.213150 20788 raft_consensus.cc:504] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.213294 20788 leader_election.cc:288] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 1 election: Requested vote from peers 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037), fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009)
I20260923 22:20:01.213480 20717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c60f297c7c40afaf9e2b86b7dda998" candidate_uuid: "7b6754039a6d4657bbca17891f115631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7d617f91a3413988462c7963e04116"
I20260923 22:20:01.213526 20583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2c60f297c7c40afaf9e2b86b7dda998" candidate_uuid: "7b6754039a6d4657bbca17891f115631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42310e349a584898ae2f3aa4ea51f4c9"
I20260923 22:20:01.213574 20717 raft_consensus.cc:3037] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:01.213644 20583 raft_consensus.cc:3037] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:01.214255 20717 raft_consensus.cc:2454] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6754039a6d4657bbca17891f115631 in term 1.
I20260923 22:20:01.214417 20583 raft_consensus.cc:2454] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6754039a6d4657bbca17891f115631 in term 1.
I20260923 22:20:01.214459 20384 leader_election.cc:302] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b6754039a6d4657bbca17891f115631, fd7d617f91a3413988462c7963e04116; no voters:
I20260923 22:20:01.214632 20788 raft_consensus.cc:2781] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:01.214763 20788 raft_consensus.cc:686] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 LEADER]: Becoming Leader. State: Replica: 7b6754039a6d4657bbca17891f115631, State: Running, Role: LEADER
I20260923 22:20:01.214825 20788 consensus_queue.cc:237] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:01.215725 20304 catalog_manager.cc:5495] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b6754039a6d4657bbca17891f115631 (127.4.207.1). New cstate: current_term: 1 leader_uuid: "7b6754039a6d4657bbca17891f115631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } health_report { overall_health: UNKNOWN } } }
I20260923 22:20:01.221472 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.219083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.4.207.1:45671)
0923 22:20:01.219137 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:01.219140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:01.219170 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:20:01.219255 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:01.219256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:01.219274 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:01.219360 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:01.219504 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.219941 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.220711 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:01.220910 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:01.220981 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:01.221160 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:01.221263 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:01.221354 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:01.221377 (+ 23us) server_negotiation.cc:299] Negotiation successful
0923 22:20:01.221396 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:20:01.223059 20717 raft_consensus.cc:1261] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 1 FOLLOWER]: Refusing update from remote peer 7b6754039a6d4657bbca17891f115631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:01.223060 20583 raft_consensus.cc:1261] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 1 FOLLOWER]: Refusing update from remote peer 7b6754039a6d4657bbca17891f115631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:01.223399 20790 consensus_queue.cc:1035] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:01.223569 20788 consensus_queue.cc:1035] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:01.227877 20304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:01.227972 20304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:01.228744 20304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Sending DeleteTablet for 3 replicas of tablet d2c60f297c7c40afaf9e2b86b7dda998
I20260923 22:20:01.229089 20429 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c60f297c7c40afaf9e2b86b7dda998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:01 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:46062
I20260923 22:20:01.229105 20563 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c60f297c7c40afaf9e2b86b7dda998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:01 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:40378
I20260923 22:20:01.229087 20697 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c60f297c7c40afaf9e2b86b7dda998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:01 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:47784
I20260923 22:20:01.229302 20805 tablet_replica.cc:330] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: stopping tablet replica
I20260923 22:20:01.229303 20804 tablet_replica.cc:330] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: stopping tablet replica
I20260923 22:20:01.229386 20805 raft_consensus.cc:2229] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:20:01.229389 20804 raft_consensus.cc:2229] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:01.229470 20805 raft_consensus.cc:2258] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:01.229485 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 20366
I20260923 22:20:01.229501 20804 raft_consensus.cc:2258] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:01.232048 20804 ts_tablet_manager.cc:1905] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:01.234092 20803 tablet_replica.cc:330] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: stopping tablet replica
I20260923 22:20:01.234197 20803 raft_consensus.cc:2229] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:01.234287 20803 raft_consensus.cc:2258] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:01.234521 20804 ts_tablet_manager.cc:1918] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:01.234678 20804 log.cc:1196] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/d2c60f297c7c40afaf9e2b86b7dda998
I20260923 22:20:01.234993 20804 ts_tablet_manager.cc:1939] T d2c60f297c7c40afaf9e2b86b7dda998 P 42310e349a584898ae2f3aa4ea51f4c9: Deleting consensus metadata
I20260923 22:20:01.235045 20803 ts_tablet_manager.cc:1905] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:01.235499 20291 catalog_manager.cc:4841] TS 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037): tablet d2c60f297c7c40afaf9e2b86b7dda998 (table test-table [id=62f7a1e91649465cacac4e0ca172269c]) successfully deleted
W20260923 22:20:01.235903 20288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:45671 (error 108)
I20260923 22:20:01.236150 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 20500
I20260923 22:20:01.236802 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:01.236330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:45671 (local address 127.0.0.1:46078)
0923 22:20:01.236730 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:01.236772 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
W20260923 22:20:01.237023 20288 catalog_manager.cc:4584] TS 7b6754039a6d4657bbca17891f115631 (127.4.207.1:45671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2c60f297c7c40afaf9e2b86b7dda998: Network error: Client connection negotiation failed: client connection to 127.4.207.1:45671: connect: Connection refused (error 111)
I20260923 22:20:01.237326 20803 ts_tablet_manager.cc:1918] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:01.237389 20803 log.cc:1196] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/d2c60f297c7c40afaf9e2b86b7dda998
I20260923 22:20:01.237586 20803 ts_tablet_manager.cc:1939] T d2c60f297c7c40afaf9e2b86b7dda998 P fd7d617f91a3413988462c7963e04116: Deleting consensus metadata
I20260923 22:20:01.238021 20290 catalog_manager.cc:4841] TS fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009): tablet d2c60f297c7c40afaf9e2b86b7dda998 (table test-table [id=62f7a1e91649465cacac4e0ca172269c]) successfully deleted
I20260923 22:20:01.241588 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 20634
I20260923 22:20:01.246430 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 20272
I20260923 22:20:21.251861 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:38315
--webserver_interface=127.4.207.62
--webserver_port=46781
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:38315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:21.332954 20807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:21.333146 20807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:21.333168 20807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:21.334229 20807 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 22:20:21.334259 20807 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 22:20:21.334272 20807 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 22:20:21.334285 20807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:21.334297 20807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:21.334311 20807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:21.334335 20807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:21.334355 20807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:21.335705 20807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:38315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:38315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=46781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:21.335909 20807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:21.336066 20807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:21.344655 20816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:21.344697 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:21.344743 20807 server_base.cc:1034] running on GCE node
W20260923 22:20:21.344684 20814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:21.345083 20807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:21.345405 20807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:21.346540 20807 hybrid_clock.cc:648] HybridClock initialized: now 1790202021346527 us; error 25 us; skew 500 ppm
Sep 23 22:20:21 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:21.353278 20807 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:20:21.353729 20807 webserver.cc:460] Webserver started at http://127.4.207.62:46781/ using document root <none> and password file <none>
I20260923 22:20:21.353962 20807 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:21.354069 20807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:21.355492 20807 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:21.356216 20823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:21.356382 20807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:21.356479 20807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "67e597cddb5a4d60a997b1d016783c43"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "80104879d318f9ccc084ed593a8ac22e"
server_key_iv: "c6a669fc66675e1658eba12034035c98"
server_key_version: "encryptionkey@0"
I20260923 22:20:21.356900 20807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:21.393512 20807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:21.395502 20807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:21.395685 20807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:21.399338 20807 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:38315
I20260923 22:20:21.399420 20875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:38315 every 8 connection(s)
I20260923 22:20:21.399823 20807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:21.401374 20876 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:20:21.402089 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Bootstrap starting.
I20260923 22:20:21.407353 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 20807
I20260923 22:20:21.407651 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:45671
--local_ip_for_outbound_sockets=127.4.207.1
--tserver_master_addrs=127.4.207.62:38315
--webserver_port=34973
--webserver_interface=127.4.207.1
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:21.411568 20876 log.cc:824] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:21.413552 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:20:21.413874 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Bootstrap complete.
I20260923 22:20:21.416460 20876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:21.416723 20876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e597cddb5a4d60a997b1d016783c43, State: Initialized, Role: FOLLOWER
I20260923 22:20:21.416914 20876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:21.417001 20876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:21.417063 20876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:21.417121 20876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 1 FOLLOWER]: Advancing to term 2
I20260923 22:20:21.418213 20876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:21.418337 20876 leader_election.cc:302] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67e597cddb5a4d60a997b1d016783c43; no voters:
I20260923 22:20:21.418545 20876 leader_election.cc:288] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 22:20:21.418582 20880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 22:20:21.418720 20880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 2 LEADER]: Becoming Leader. State: Replica: 67e597cddb5a4d60a997b1d016783c43, State: Running, Role: LEADER
I20260923 22:20:21.418834 20876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:21.418874 20880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:21.419198 20881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "67e597cddb5a4d60a997b1d016783c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } } }
I20260923 22:20:21.419354 20881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:21.419195 20880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67e597cddb5a4d60a997b1d016783c43. Latest consensus state: current_term: 2 leader_uuid: "67e597cddb5a4d60a997b1d016783c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } } }
I20260923 22:20:21.419484 20880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:21.420269 20888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:21.421062 20888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:21.421217 20888 catalog_manager.cc:1259] Loaded cluster ID: a63892d414ff43768252e0dea078e323
I20260923 22:20:21.421283 20888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:20:21.422192 20888 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:21.422398 20888 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Loaded TSK: 0
I20260923 22:20:21.432687 20888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Generated new TSK 1
I20260923 22:20:21.432806 20888 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:20:21.498838 20878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:21.499045 20878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:21.499084 20878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:21.500254 20878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:21.500319 20878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:21.500363 20878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:20:21.501711 20878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:45671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=34973
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:21.501945 20878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:21.502158 20878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:21.507902 20906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:21.507936 20904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:21.507988 20878 server_base.cc:1034] running on GCE node
W20260923 22:20:21.507910 20903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:21.508360 20878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:21.508641 20878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:21.509810 20878 hybrid_clock.cc:648] HybridClock initialized: now 1790202021509782 us; error 45 us; skew 500 ppm
Sep 23 22:20:21 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:21.512343 20878 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:20:21.512817 20878 webserver.cc:460] Webserver started at http://127.4.207.1:34973/ using document root <none> and password file <none>
I20260923 22:20:21.513060 20878 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:21.513132 20878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:21.514740 20878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:20:21.515527 20913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:21.515744 20878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:21.515826 20878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "7b6754039a6d4657bbca17891f115631"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "ec99ecc1f43fb73b4edebc4543915d96"
server_key_iv: "352cfbba9c0d58d8286b9006228decad"
server_key_version: "encryptionkey@0"
I20260923 22:20:21.516225 20878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:21.557989 20878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:21.560318 20878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:21.560443 20878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:21.560662 20878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:21.561064 20920 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:20:21.562495 20878 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:20:21.562551 20878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 22:20:21.562580 20878 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:20:21.563335 20878 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:20:21.563373 20878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:20:21.563472 20920 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Bootstrap starting.
I20260923 22:20:21.570322 20878 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:45671
I20260923 22:20:21.570389 21027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:45671 every 8 connection(s)
I20260923 22:20:21.570698 20878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:21.573388 20920 log.cc:824] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:21.573495 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 20878
I20260923 22:20:21.573791 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:39037
--local_ip_for_outbound_sockets=127.4.207.2
--tserver_master_addrs=127.4.207.62:38315
--webserver_port=34881
--webserver_interface=127.4.207.2
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:21.576062 20920 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:20:21.576368 20920 tablet_bootstrap.cc:492] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Bootstrap complete.
I20260923 22:20:21.577292 20920 ts_tablet_manager.cc:1397] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260923 22:20:21.578480 20920 raft_consensus.cc:348] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:21.578838 20920 raft_consensus.cc:729] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6754039a6d4657bbca17891f115631, State: Initialized, Role: FOLLOWER
I20260923 22:20:21.578975 20920 consensus_queue.cc:260] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } }
I20260923 22:20:21.579226 20920 ts_tablet_manager.cc:1428] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 23 22:20:21 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:21.584812 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.571359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:55693)
0923 22:20:21.571563 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.571608 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:21.572362 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:21.574137 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:21.574141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:21.574487 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:21.574776 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.574782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.576404 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.576405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.577844 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.577847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.578635 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.578636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.578799 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.579419 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:21.579427 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:21.581004 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:21.583117 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:21.583123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:21.583124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:21.583320 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:21.583686 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:21.583687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:21.583687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:21.583794 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:21.584331 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:21.584336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:21.584446 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 22:20:21.584530 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 22:20:21.584648 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":126,"threads_started":1}
I20260923 22:20:21.584957 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.571509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:55693 (local address 127.4.207.62:38315)
0923 22:20:21.573671 (+ 2162us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.573690 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.573730 (+ 40us) server_negotiation.cc:407] Connection header received
0923 22:20:21.573861 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.573868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.573927 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.574081 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:21.574920 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.576256 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.577939 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.578532 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.578651 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.581108 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:21.581112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:21.581112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:21.581124 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:21.583041 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.583418 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.583421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.583422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.583560 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.583880 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.583881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.583881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.584255 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:21.584347 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.584619 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.584771 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2110,"thread_start_us":78,"threads_started":1}
I20260923 22:20:21.585558 21028 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38315
I20260923 22:20:21.585657 21028 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:21.585812 21028 heartbeater.cc:511] Master 127.4.207.62:38315 requested a full tablet report, sending...
I20260923 22:20:21.586486 20840 ts_manager.cc:194] Registered new tserver with Master: 7b6754039a6d4657bbca17891f115631 (127.4.207.1:45671)
I20260923 22:20:21.587570 20840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:55693
I20260923 22:20:21.588372 21028 heartbeater.cc:503] Master 127.4.207.62:38315 was elected leader, sending a full tablet report...
Sep 23 22:20:21 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.1@KRBTEST.COM
I20260923 22:20:21.595033 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.587125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37242 (local address 127.4.207.1:45671)
0923 22:20:21.587271 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.587275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.587362 (+ 87us) server_negotiation.cc:407] Connection header received
0923 22:20:21.587427 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.587429 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.587470 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.587539 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:21.588015 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.589114 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.589851 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.590287 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.590378 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.592387 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:21.592393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:21.592394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:21.592399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:21.593887 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.594313 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.594315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.594316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.594364 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.594548 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.594549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.594550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.594668 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:21.594734 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.594903 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.594976 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260923 22:20:21.595075 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.587051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:45671 (local address 127.0.0.1:37242)
0923 22:20:21.587258 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.587293 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:21.587361 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:21.587553 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:21.587556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:21.587883 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:21.587937 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.587939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.589201 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.589202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.589778 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.589780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.590384 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.590385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.590435 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.590857 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:21.590861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:21.592310 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:21.593970 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:21.593977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:21.593978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:21.594235 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:21.594427 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:21.594428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:21.594429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:21.594490 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:21.594730 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:21.594736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:21.594786 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 22:20:21.594868 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:20:21.594977 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260923 22:20:21.595510 20962 tablet_service.cc:1514] Processing DeleteTablet for tablet d2c60f297c7c40afaf9e2b86b7dda998 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:01 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:37242
I20260923 22:20:21.596068 21037 tablet_replica.cc:330] stopping tablet replica
I20260923 22:20:21.596169 21037 raft_consensus.cc:2229] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:21.596238 21037 raft_consensus.cc:2258] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:21.596510 21037 ts_tablet_manager.cc:1905] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:21.598623 21037 ts_tablet_manager.cc:1918] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:21.598726 21037 log.cc:1196] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal/wals/d2c60f297c7c40afaf9e2b86b7dda998
I20260923 22:20:21.598949 21037 ts_tablet_manager.cc:1939] T d2c60f297c7c40afaf9e2b86b7dda998 P 7b6754039a6d4657bbca17891f115631: Deleting consensus metadata
I20260923 22:20:21.599426 20824 catalog_manager.cc:4841] TS 7b6754039a6d4657bbca17891f115631 (127.4.207.1:45671): tablet d2c60f297c7c40afaf9e2b86b7dda998 (table test-table [id=62f7a1e91649465cacac4e0ca172269c]) successfully deleted
W20260923 22:20:21.659368 21033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:21.659618 21033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:21.659653 21033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:21.660776 21033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:21.660815 21033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:21.660838 21033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:20:21.662070 21033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:39037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=34881
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:21.662253 21033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:21.662420 21033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:21.667229 21042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:21.667276 21045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:21.667326 21043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:21.667474 21033 server_base.cc:1034] running on GCE node
I20260923 22:20:21.667646 21033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:21.667868 21033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:21.669009 21033 hybrid_clock.cc:648] HybridClock initialized: now 1790202021668975 us; error 49 us; skew 500 ppm
Sep 23 22:20:21 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:21.671427 21033 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:20:21.671902 21033 webserver.cc:460] Webserver started at http://127.4.207.2:34881/ using document root <none> and password file <none>
I20260923 22:20:21.672094 21033 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:21.672147 21033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:21.673337 21033 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:21.673933 21052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:21.674100 21033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:20:21.674192 21033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "42310e349a584898ae2f3aa4ea51f4c9"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "0389cf18a71353c203219f6c9cd6c621"
server_key_iv: "5301ad36f643e7352c6506137875d657"
server_key_version: "encryptionkey@0"
I20260923 22:20:21.674522 21033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:21.684208 21033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:21.686066 21033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:21.686223 21033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:21.686432 21033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:21.686719 21033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:21.686772 21033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:21.686810 21033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:21.686836 21033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:21.692399 21033 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:39037
I20260923 22:20:21.692492 21165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:39037 every 8 connection(s)
I20260923 22:20:21.692791 21033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
Sep 23 22:20:21 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:21.698961 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 21033
I20260923 22:20:21.699265 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:33009
--local_ip_for_outbound_sockets=127.4.207.3
--tserver_master_addrs=127.4.207.62:38315
--webserver_port=42551
--webserver_interface=127.4.207.3
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:21.702183 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.693495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:54245)
0923 22:20:21.693693 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.693728 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:21.694182 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:21.694405 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:21.694409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:21.694984 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:21.695162 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.695168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.695920 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.695921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.696719 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.696721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.697084 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.697085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.697178 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.697604 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:21.697611 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:21.699190 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:21.700893 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:21.700899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:21.700900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:21.701104 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:21.701316 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:21.701316 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:21.701317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:21.701407 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:21.701676 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:21.701680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:21.701783 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 22:20:21.701900 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 22:20:21.702013 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260923 22:20:21.702210 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.693503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:54245 (local address 127.4.207.62:38315)
0923 22:20:21.693566 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.693569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.693818 (+ 249us) server_negotiation.cc:407] Connection header received
0923 22:20:21.694264 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.694265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.694278 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.694331 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:21.695253 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.695803 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.696805 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.697011 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.697064 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.699279 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:21.699281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:21.699281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:21.699287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:21.700809 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.701183 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.701184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.701185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.701248 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.701468 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.701468 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.701469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.701612 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:21.701682 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.701946 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.702037 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:20:21.702657 21166 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38315
I20260923 22:20:21.702762 21166 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:21.702932 21166 heartbeater.cc:511] Master 127.4.207.62:38315 requested a full tablet report, sending...
I20260923 22:20:21.703325 20840 ts_manager.cc:194] Registered new tserver with Master: 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037)
I20260923 22:20:21.703907 20840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:54245
W20260923 22:20:21.779265 21169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:21.779484 21169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:21.779508 21169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:21.780584 21169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:21.780622 21169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:21.780647 21169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:20:21.781989 21169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:33009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=42551
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:21.782177 21169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:21.782346 21169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:21.787424 21174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:21.787420 21177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:21.787635 21169 server_base.cc:1034] running on GCE node
W20260923 22:20:21.787477 21175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:21.787992 21169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:21.788228 21169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:21.789376 21169 hybrid_clock.cc:648] HybridClock initialized: now 1790202021789355 us; error 37 us; skew 500 ppm
Sep 23 22:20:21 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:21.791821 21169 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:20:21.792294 21169 webserver.cc:460] Webserver started at http://127.4.207.3:42551/ using document root <none> and password file <none>
I20260923 22:20:21.792538 21169 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:21.792609 21169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:21.793951 21169 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:21.794536 21184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:21.794723 21169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:20:21.794824 21169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "fd7d617f91a3413988462c7963e04116"
format_stamp: "Formatted at 2026-09-23 22:20:01 on dist-test-slave-7511"
server_key: "c458d45aadf41bb74e168f66c8715b1c"
server_key_iv: "b680c30590dffde97f7c16b4bddd694b"
server_key_version: "encryptionkey@0"
I20260923 22:20:21.795162 21169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:21.802989 21169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:21.804857 21169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:21.805011 21169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:21.805217 21169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:21.805505 21169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:21.805557 21169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:21.805595 21169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:21.805621 21169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:21.811584 21169 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:33009
I20260923 22:20:21.811630 21297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:33009 every 8 connection(s)
I20260923 22:20:21.811945 21169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:20:21.814527 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 21169
Sep 23 22:20:21 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202021, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:21.822227 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.814967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.4.207.62:38315)
0923 22:20:21.815077 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.815079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.815182 (+ 103us) server_negotiation.cc:407] Connection header received
0923 22:20:21.815222 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.815223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.815235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.815314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:21.815940 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.816596 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.817390 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.817697 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.817778 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.819207 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:21.819208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:21.819208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:21.819215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:21.821031 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.821454 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.821455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.821455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.821509 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.821713 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.821714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.821714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.821832 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:21.821998 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.822035 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.822112 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260923 22:20:21.822427 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.812575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:40191 (local address 127.4.207.62:38315)
0923 22:20:21.812640 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.812642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.812850 (+ 208us) server_negotiation.cc:407] Connection header received
0923 22:20:21.813355 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.813357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.813370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.813421 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:21.814365 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.814998 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.816417 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.816610 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.816729 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.819320 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:21.819322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:21.819322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:21.819329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:21.821136 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.821506 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.821507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.821508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.821560 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:21.821824 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:21.821825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:21.821825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:21.821930 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:21.821998 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.822267 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.822342 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:20:21.822625 21300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.812546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:40191)
0923 22:20:21.812733 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.812767 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:21.813250 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:21.813503 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:21.813507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:21.814077 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:21.814263 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.814267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.815097 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.815098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.816282 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.816285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.816745 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.816746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.816842 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.817356 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:21.817368 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:21.819217 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:21.821202 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:21.821208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:21.821210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:21.821430 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:21.821637 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:21.821638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:21.821638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:21.821750 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:21.821992 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:21.821998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:21.822133 (+ 135us) client_negotiation.cc:769] Sending connection context
0923 22:20:21.822246 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 22:20:21.822398 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260923 22:20:21.823019 21298 heartbeater.cc:347] Connected to a master server at 127.4.207.62:38315
I20260923 22:20:21.823093 21298 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:21.823235 21298 heartbeater.cc:511] Master 127.4.207.62:38315 requested a full tablet report, sending...
I20260923 22:20:21.823576 20840 ts_manager.cc:194] Registered new tserver with Master: fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009)
I20260923 22:20:21.823956 20840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:40191
I20260923 22:20:21.833935 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:20:21.836946 21302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.834296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.4.207.62:38315)
0923 22:20:21.834370 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.834372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.834485 (+ 113us) server_negotiation.cc:407] Connection header received
0923 22:20:21.834576 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.834577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.834630 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.834733 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:21.834871 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.835360 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.836149 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.836370 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.836444 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.836625 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:21.836735 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:21.836828 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.836846 (+ 18us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.836867 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:20:21.837241 20840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:20:21.838832 20840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:20:21.841104 20962 tablet_service.cc:1467] Processing CreateTablet for tablet 5f94b496273e4e84bf589d5310733841 (DEFAULT_TABLE table=test-table [id=f284dbcf4edb47dbb10c356678cdcd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:21.841323 20962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f94b496273e4e84bf589d5310733841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:21.842983 20920 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Bootstrap starting.
I20260923 22:20:21.843277 20920 tablet_bootstrap.cc:654] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:21.843786 20920 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: No bootstrap required, opened a new log
I20260923 22:20:21.843849 20920 ts_tablet_manager.cc:1397] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:21.844255 20920 raft_consensus.cc:348] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.844338 20920 raft_consensus.cc:374] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:21.844384 20920 raft_consensus.cc:729] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6754039a6d4657bbca17891f115631, State: Initialized, Role: FOLLOWER
I20260923 22:20:21.844432 20920 consensus_queue.cc:260] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.844513 20920 ts_tablet_manager.cc:1428] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:21.844599 21035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.840782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33009 (local address 127.0.0.1:53544)
0923 22:20:21.840815 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.840828 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:21.840866 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:21.841512 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:21.841513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:21.841515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:21.841580 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.841582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.842415 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.842416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.843493 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.843496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.844352 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.844353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.844417 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.844435 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 22:20:21.844486 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:20:21.844530 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":16}
I20260923 22:20:21.844609 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.841096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.4.207.3:33009)
0923 22:20:21.841324 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.841328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.841338 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:20:21.841378 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.841380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.841397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.841562 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:21.841643 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.842328 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.843581 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.844273 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.844353 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.844411 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.844499 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.844535 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":120,"threads_started":1}
I20260923 22:20:21.845299 21232 tablet_service.cc:1467] Processing CreateTablet for tablet 5f94b496273e4e84bf589d5310733841 (DEFAULT_TABLE table=test-table [id=f284dbcf4edb47dbb10c356678cdcd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:21.845602 21232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f94b496273e4e84bf589d5310733841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:21.845738 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.840855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:39037 (local address 127.0.0.1:52650)
0923 22:20:21.840994 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.841005 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:21.841051 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:21.841519 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:21.841520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:21.841522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:21.841654 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.841655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.842963 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.842963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.843986 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.843988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.845512 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.845513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.845575 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.845579 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:20:21.845634 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 22:20:21.845672 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260923 22:20:21.845760 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.841102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.4.207.2:39037)
0923 22:20:21.841311 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.841317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.841328 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:20:21.841411 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.841414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.841430 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.841516 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:21.841711 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.842878 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.844073 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.845406 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.845506 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.845560 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.845647 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.845688 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
I20260923 22:20:21.846298 21100 tablet_service.cc:1467] Processing CreateTablet for tablet 5f94b496273e4e84bf589d5310733841 (DEFAULT_TABLE table=test-table [id=f284dbcf4edb47dbb10c356678cdcd92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:21.846556 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f94b496273e4e84bf589d5310733841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:21.848453 21308 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: Bootstrap starting.
I20260923 22:20:21.848973 21308 tablet_bootstrap.cc:654] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:21.849360 21309 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: Bootstrap starting.
I20260923 22:20:21.849574 21308 log.cc:824] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:21.850050 21309 tablet_bootstrap.cc:654] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:21.850342 21308 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: No bootstrap required, opened a new log
I20260923 22:20:21.850428 21308 ts_tablet_manager.cc:1397] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:20:21.850638 21309 log.cc:824] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:21.851366 21309 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: No bootstrap required, opened a new log
I20260923 22:20:21.851457 21309 ts_tablet_manager.cc:1397] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:21.852370 21308 raft_consensus.cc:348] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.852540 21308 raft_consensus.cc:374] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:21.852578 21308 raft_consensus.cc:729] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7d617f91a3413988462c7963e04116, State: Initialized, Role: FOLLOWER
I20260923 22:20:21.852705 21308 consensus_queue.cc:260] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.852973 21308 ts_tablet_manager.cc:1428] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:20:21.852994 21298 heartbeater.cc:503] Master 127.4.207.62:38315 was elected leader, sending a full tablet report...
I20260923 22:20:21.853287 21309 raft_consensus.cc:348] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.853407 21309 raft_consensus.cc:374] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:21.853451 21309 raft_consensus.cc:729] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42310e349a584898ae2f3aa4ea51f4c9, State: Initialized, Role: FOLLOWER
I20260923 22:20:21.853574 21309 consensus_queue.cc:260] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.853861 21309 ts_tablet_manager.cc:1428] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 22:20:21.853883 21166 heartbeater.cc:503] Master 127.4.207.62:38315 was elected leader, sending a full tablet report...
I20260923 22:20:21.888900 21034 raft_consensus.cc:482] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:21.889027 21034 raft_consensus.cc:504] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.889349 21034 leader_election.cc:288] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009), 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037)
I20260923 22:20:21.892227 21305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.889632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:33463 (local address 127.4.207.3:33009)
0923 22:20:21.889682 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.889685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.889693 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:21.889718 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.889719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.889730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.889786 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:21.889928 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.890417 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.891354 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.891862 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.891943 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.891953 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.892117 (+ 164us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.892151 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 22:20:21.892227 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.889494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33009 (local address 127.4.207.1:33463)
0923 22:20:21.889571 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.889583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:21.889636 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:21.889789 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:21.889790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:21.889791 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:21.889849 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.889851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.890506 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.890507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.891262 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.891264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.891941 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.891942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.892021 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.892041 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 22:20:21.892111 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:20:21.892152 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 22:20:21.892397 21306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.889641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:55551 (local address 127.4.207.2:39037)
0923 22:20:21.889728 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.889730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.889806 (+ 76us) server_negotiation.cc:407] Connection header received
0923 22:20:21.889896 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.889897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.889909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.889986 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:21.890147 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.890590 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.891641 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.892033 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.892115 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.892124 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.892293 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.892328 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 22:20:21.892416 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.889527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:39037 (local address 127.4.207.1:55551)
0923 22:20:21.889721 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.889733 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:21.889822 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:21.889989 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:21.889990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:21.889991 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:21.890069 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.890071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.890699 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.890700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.891561 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:21.891564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.892143 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.892145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:21.892219 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.892222 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:21.892289 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 22:20:21.892327 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260923 22:20:21.892575 21252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f94b496273e4e84bf589d5310733841" candidate_uuid: "7b6754039a6d4657bbca17891f115631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7d617f91a3413988462c7963e04116" is_pre_election: true
I20260923 22:20:21.892758 21252 raft_consensus.cc:2454] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6754039a6d4657bbca17891f115631 in term 0.
I20260923 22:20:21.892737 21120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f94b496273e4e84bf589d5310733841" candidate_uuid: "7b6754039a6d4657bbca17891f115631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42310e349a584898ae2f3aa4ea51f4c9" is_pre_election: true
I20260923 22:20:21.892858 21120 raft_consensus.cc:2454] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b6754039a6d4657bbca17891f115631 in term 0.
I20260923 22:20:21.893004 20916 leader_election.cc:302] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b6754039a6d4657bbca17891f115631, fd7d617f91a3413988462c7963e04116; no voters:
I20260923 22:20:21.893146 21034 raft_consensus.cc:2781] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:21.893210 21034 raft_consensus.cc:482] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:21.893227 21034 raft_consensus.cc:3037] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:21.894079 21034 raft_consensus.cc:504] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.894263 21034 leader_election.cc:288] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 1 election: Requested vote from peers fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009), 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037)
I20260923 22:20:21.894430 21120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f94b496273e4e84bf589d5310733841" candidate_uuid: "7b6754039a6d4657bbca17891f115631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42310e349a584898ae2f3aa4ea51f4c9"
I20260923 22:20:21.894413 21252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f94b496273e4e84bf589d5310733841" candidate_uuid: "7b6754039a6d4657bbca17891f115631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd7d617f91a3413988462c7963e04116"
I20260923 22:20:21.894500 21120 raft_consensus.cc:3037] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:21.894500 21252 raft_consensus.cc:3037] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:21.895397 21120 raft_consensus.cc:2454] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6754039a6d4657bbca17891f115631 in term 1.
I20260923 22:20:21.895397 21252 raft_consensus.cc:2454] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b6754039a6d4657bbca17891f115631 in term 1.
I20260923 22:20:21.895614 20917 leader_election.cc:302] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42310e349a584898ae2f3aa4ea51f4c9, 7b6754039a6d4657bbca17891f115631; no voters:
I20260923 22:20:21.895776 21034 raft_consensus.cc:2781] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:21.895941 21034 raft_consensus.cc:686] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 LEADER]: Becoming Leader. State: Replica: 7b6754039a6d4657bbca17891f115631, State: Running, Role: LEADER
I20260923 22:20:21.896020 21034 consensus_queue.cc:237] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:21.896786 20840 catalog_manager.cc:5495] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b6754039a6d4657bbca17891f115631 (127.4.207.1). New cstate: current_term: 1 leader_uuid: "7b6754039a6d4657bbca17891f115631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } health_report { overall_health: HEALTHY } } }
I20260923 22:20:21.903565 21036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.901207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.4.207.1:45671)
0923 22:20:21.901262 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:21.901264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:21.901273 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:20:21.901347 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:21.901347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:21.901364 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:21.901450 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:21.901572 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.902020 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.902756 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:21.902975 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:21.903046 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:21.903226 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:21.903330 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:21.903412 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:21.903470 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 22:20:21.903494 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:20:21.904989 21120 raft_consensus.cc:1261] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 1 FOLLOWER]: Refusing update from remote peer 7b6754039a6d4657bbca17891f115631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:21.905010 21252 raft_consensus.cc:1261] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 1 FOLLOWER]: Refusing update from remote peer 7b6754039a6d4657bbca17891f115631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:21.905332 21315 consensus_queue.cc:1035] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:21.905472 21034 consensus_queue.cc:1035] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:21.911078 20839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:21.911171 20839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:21.911934 20839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Sending DeleteTablet for 3 replicas of tablet 5f94b496273e4e84bf589d5310733841
I20260923 22:20:21.912267 20962 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f94b496273e4e84bf589d5310733841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:21 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:37242
I20260923 22:20:21.912271 21232 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f94b496273e4e84bf589d5310733841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:21 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:53544
I20260923 22:20:21.912418 21100 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f94b496273e4e84bf589d5310733841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:21 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:52650
I20260923 22:20:21.912456 21037 tablet_replica.cc:330] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: stopping tablet replica
I20260923 22:20:21.912519 21037 raft_consensus.cc:2229] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:20:21.912616 21037 raft_consensus.cc:2258] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:21.912633 21327 tablet_replica.cc:330] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: stopping tablet replica
I20260923 22:20:21.912757 21327 raft_consensus.cc:2229] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:21.912869 21327 raft_consensus.cc:2258] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:21.912989 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 20878
I20260923 22:20:21.913019 21328 tablet_replica.cc:330] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: stopping tablet replica
I20260923 22:20:21.913100 21328 raft_consensus.cc:2229] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:21.913189 21328 raft_consensus.cc:2258] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:21.914099 21327 ts_tablet_manager.cc:1905] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:21.914135 21328 ts_tablet_manager.cc:1905] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:21.916045 21327 ts_tablet_manager.cc:1918] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:21.916119 21327 log.cc:1196] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/5f94b496273e4e84bf589d5310733841
I20260923 22:20:21.916124 21328 ts_tablet_manager.cc:1918] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:21.916198 21328 log.cc:1196] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/5f94b496273e4e84bf589d5310733841
I20260923 22:20:21.916399 21327 ts_tablet_manager.cc:1939] T 5f94b496273e4e84bf589d5310733841 P fd7d617f91a3413988462c7963e04116: Deleting consensus metadata
I20260923 22:20:21.916505 21328 ts_tablet_manager.cc:1939] T 5f94b496273e4e84bf589d5310733841 P 42310e349a584898ae2f3aa4ea51f4c9: Deleting consensus metadata
I20260923 22:20:21.916836 20826 catalog_manager.cc:4841] TS fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009): tablet 5f94b496273e4e84bf589d5310733841 (table test-table [id=f284dbcf4edb47dbb10c356678cdcd92]) successfully deleted
I20260923 22:20:21.916922 20827 catalog_manager.cc:4841] TS 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037): tablet 5f94b496273e4e84bf589d5310733841 (table test-table [id=f284dbcf4edb47dbb10c356678cdcd92]) successfully deleted
W20260923 22:20:21.918660 20824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:45671 (error 108)
I20260923 22:20:21.919130 21304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:21.919010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:45671 (local address 127.0.0.1:37264)
0923 22:20:21.919061 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:21.919095 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:20:21.919235 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 21033
W20260923 22:20:21.919303 20824 catalog_manager.cc:4584] TS 7b6754039a6d4657bbca17891f115631 (127.4.207.1:45671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f94b496273e4e84bf589d5310733841: Network error: Client connection negotiation failed: client connection to 127.4.207.1:45671: connect: Connection refused (error 111)
I20260923 22:20:21.923568 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 21169
I20260923 22:20:21.928277 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 20807
I20260923 22:20:41.933959 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:38315
--webserver_interface=127.4.207.62
--webserver_port=46781
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:38315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:42.018247 21330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:42.018468 21330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:42.018492 21330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:42.019738 21330 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 22:20:42.019778 21330 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 22:20:42.019791 21330 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 22:20:42.019804 21330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:42.019816 21330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:42.019829 21330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:42.019842 21330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:42.019855 21330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:42.021247 21330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:38315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:38315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=46781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:42.021433 21330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:42.021597 21330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:42.026289 21335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:42.026290 21336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:42.026399 21330 server_base.cc:1034] running on GCE node
W20260923 22:20:42.026381 21338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:42.026764 21330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:42.027070 21330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:42.028234 21330 hybrid_clock.cc:648] HybridClock initialized: now 1790202042028211 us; error 34 us; skew 500 ppm
Sep 23 22:20:42 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:42.033938 21330 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:20:42.034380 21330 webserver.cc:460] Webserver started at http://127.4.207.62:46781/ using document root <none> and password file <none>
I20260923 22:20:42.034610 21330 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:42.034690 21330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:42.036155 21330 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:42.036865 21345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:42.037078 21330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:42.037186 21330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "67e597cddb5a4d60a997b1d016783c43"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "80104879d318f9ccc084ed593a8ac22e"
server_key_iv: "c6a669fc66675e1658eba12034035c98"
server_key_version: "encryptionkey@0"
I20260923 22:20:42.037535 21330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:42.055577 21330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:42.057601 21330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:42.057787 21330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:42.063292 21330 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:38315
I20260923 22:20:42.063390 21397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:38315 every 8 connection(s)
I20260923 22:20:42.063812 21330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:42.065201 21398 sys_catalog.cc:263] Verifying existing consensus state
I20260923 22:20:42.065932 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Bootstrap starting.
I20260923 22:20:42.069588 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 21330
I20260923 22:20:42.069957 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:45671
--local_ip_for_outbound_sockets=127.4.207.1
--tserver_master_addrs=127.4.207.62:38315
--webserver_port=34973
--webserver_interface=127.4.207.1
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:42.075850 21398 log.cc:824] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:42.078048 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:20:42.078365 21398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: Bootstrap complete.
I20260923 22:20:42.080678 21398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:42.080899 21398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67e597cddb5a4d60a997b1d016783c43, State: Initialized, Role: FOLLOWER
I20260923 22:20:42.081066 21398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:42.081125 21398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:42.081187 21398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:42.081240 21398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 2 FOLLOWER]: Advancing to term 3
I20260923 22:20:42.082264 21398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:42.082383 21398 leader_election.cc:302] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67e597cddb5a4d60a997b1d016783c43; no voters:
I20260923 22:20:42.082589 21398 leader_election.cc:288] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 22:20:42.082625 21402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 22:20:42.082746 21402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [term 3 LEADER]: Becoming Leader. State: Replica: 67e597cddb5a4d60a997b1d016783c43, State: Running, Role: LEADER
I20260923 22:20:42.082860 21398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:42.082860 21402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } }
I20260923 22:20:42.083165 21404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67e597cddb5a4d60a997b1d016783c43. Latest consensus state: current_term: 3 leader_uuid: "67e597cddb5a4d60a997b1d016783c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } } }
I20260923 22:20:42.083168 21403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "67e597cddb5a4d60a997b1d016783c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67e597cddb5a4d60a997b1d016783c43" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 38315 } } }
I20260923 22:20:42.083355 21403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:42.083986 21404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:42.084805 21418 catalog_manager.cc:1259] Loaded cluster ID: a63892d414ff43768252e0dea078e323
I20260923 22:20:42.084882 21418 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: loading cluster ID for follower catalog manager: success
I20260923 22:20:42.086068 21418 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67e597cddb5a4d60a997b1d016783c43: acquiring CA information for follower catalog manager: success
F20260923 22:20:42.086437 21418 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fecb8f8c9ad google::LogMessage::Fail() at ??:0
@ 0x7fecb8f8f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fecb8f8c4de google::LogMessage::Flush() at ??:0
@ 0x7fecb8f8da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fecbb6f9104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fecc0ee96b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fecc0ee9989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fecc0ee9e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fecc0eec8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fecbae4f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fecbbcb36db start_thread at ??:0
@ 0x7fecb7a8a61f clone at ??:0
W20260923 22:20:42.166231 21400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:42.166430 21400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:42.166458 21400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:42.167676 21400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:42.167716 21400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:42.167742 21400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:20:42.169170 21400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:45671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=34973
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:42.169405 21400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:42.169646 21400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:42.174594 21436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:42.174659 21437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:42.174629 21439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:42.174796 21400 server_base.cc:1034] running on GCE node
I20260923 22:20:42.175052 21400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:42.175325 21400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:42.176484 21400 hybrid_clock.cc:648] HybridClock initialized: now 1790202042176476 us; error 47 us; skew 500 ppm
Sep 23 22:20:42 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:42.179009 21400 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:20:42.179492 21400 webserver.cc:460] Webserver started at http://127.4.207.1:34973/ using document root <none> and password file <none>
I20260923 22:20:42.179723 21400 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:42.179766 21400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:42.181057 21400 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:42.181746 21446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:42.181929 21400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:42.181998 21400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "7b6754039a6d4657bbca17891f115631"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "ec99ecc1f43fb73b4edebc4543915d96"
server_key_iv: "352cfbba9c0d58d8286b9006228decad"
server_key_version: "encryptionkey@0"
I20260923 22:20:42.182351 21400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:42.191148 21400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:42.193189 21400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:42.193328 21400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:42.193552 21400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:42.193959 21453 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 22:20:42.195876 21400 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 22:20:42.195937 21400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 22:20:42.196022 21400 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 22:20:42.196606 21400 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 22:20:42.196673 21400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 22:20:42.196766 21453 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Bootstrap starting.
I20260923 22:20:42.203620 21400 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:45671
I20260923 22:20:42.203660 21560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:45671 every 8 connection(s)
I20260923 22:20:42.204007 21400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:42.206889 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 21400
I20260923 22:20:42.207173 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:39037
--local_ip_for_outbound_sockets=127.4.207.2
--tserver_master_addrs=127.4.207.62:38315
--webserver_port=34881
--webserver_interface=127.4.207.2
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:20:42.207358 21453 log.cc:824] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:42.210255 21453 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 22:20:42.210565 21453 tablet_bootstrap.cc:492] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Bootstrap complete.
I20260923 22:20:42.211799 21453 ts_tablet_manager.cc:1397] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.007s
I20260923 22:20:42.213613 21453 raft_consensus.cc:348] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:42.214005 21453 raft_consensus.cc:729] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b6754039a6d4657bbca17891f115631, State: Initialized, Role: FOLLOWER
I20260923 22:20:42.214162 21453 consensus_queue.cc:260] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:42.214480 21453 ts_tablet_manager.cc:1428] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260923 22:20:42.272228 21563 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:20:42.204720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:46187)
0923 22:20:42.204889 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.204931 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:42.205362 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:42.271999 (+ 66637us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
W20260923 22:20:42.272527 21561 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:38315 (0 consecutive failures): Network error: Failed to ping master at 127.4.207.62:38315: Client connection negotiation failed: client connection to 127.4.207.62:38315: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 22:20:42.272863 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.272741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:50149)
0923 22:20:42.272798 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.272842 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 22:20:42.273211 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.273126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:49085)
0923 22:20:42.273173 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.273196 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 22:20:42.273337 21561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 22:20:42.290284 21565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:42.290480 21565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:42.290504 21565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:42.291718 21565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:42.291759 21565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:42.291806 21565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:20:42.293336 21565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:39037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=34881
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:42.293579 21565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:42.293810 21565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:42.298897 21572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:42.298957 21571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:42.298945 21565 server_base.cc:1034] running on GCE node
W20260923 22:20:42.298908 21574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:42.299293 21565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:42.299578 21565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:42.300770 21565 hybrid_clock.cc:648] HybridClock initialized: now 1790202042300756 us; error 29 us; skew 500 ppm
Sep 23 22:20:42 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:42.303506 21565 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:20:42.304039 21565 webserver.cc:460] Webserver started at http://127.4.207.2:34881/ using document root <none> and password file <none>
I20260923 22:20:42.304286 21565 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:42.304361 21565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:42.305712 21565 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:42.306294 21581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:42.306468 21565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:20:42.306572 21565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "42310e349a584898ae2f3aa4ea51f4c9"
format_stamp: "Formatted at 2026-09-23 22:20:00 on dist-test-slave-7511"
server_key: "0389cf18a71353c203219f6c9cd6c621"
server_key_iv: "5301ad36f643e7352c6506137875d657"
server_key_version: "encryptionkey@0"
I20260923 22:20:42.306907 21565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:42.326495 21565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:42.328970 21565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:42.329137 21565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:42.329380 21565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:42.329689 21565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:42.329742 21565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:42.329778 21565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:42.329807 21565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:42.335850 21565 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:39037
I20260923 22:20:42.335914 21694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:39037 every 8 connection(s)
I20260923 22:20:42.336230 21565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:20:42.337273 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.336904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:53681)
0923 22:20:42.337056 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.337135 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
W20260923 22:20:42.337664 21695 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:38315 (0 consecutive failures): Network error: Failed to ping master at 127.4.207.62:38315: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
I20260923 22:20:42.337985 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.337895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:55039)
0923 22:20:42.337942 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.337968 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 22:20:42.338268 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.338218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:33981)
0923 22:20:42.338239 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.338256 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260923 22:20:42.338392 21695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:20:42.343302 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 21565
I20260923 22:20:42.343636 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:33009
--local_ip_for_outbound_sockets=127.4.207.3
--tserver_master_addrs=127.4.207.62:38315
--webserver_port=42551
--webserver_interface=127.4.207.3
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:42.426668 21698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:42.426888 21698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:42.426914 21698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:42.427995 21698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:42.428033 21698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:42.428059 21698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:20:42.429412 21698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:33009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=42551
--tserver_master_addrs=127.4.207.62:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:42.429603 21698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:42.429797 21698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:42.434777 21706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:42.434784 21703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:42.434830 21698 server_base.cc:1034] running on GCE node
W20260923 22:20:42.434777 21704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:42.435148 21698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:42.435374 21698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:42.436533 21698 hybrid_clock.cc:648] HybridClock initialized: now 1790202042436523 us; error 31 us; skew 500 ppm
Sep 23 22:20:42 dist-test-slave-7511 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:42.439055 21698 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:20:42.439510 21698 webserver.cc:460] Webserver started at http://127.4.207.3:42551/ using document root <none> and password file <none>
I20260923 22:20:42.439735 21698 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:42.439800 21698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:42.441087 21698 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:42.441707 21713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:42.441872 21698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:42.441965 21698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "fd7d617f91a3413988462c7963e04116"
format_stamp: "Formatted at 2026-09-23 22:20:01 on dist-test-slave-7511"
server_key: "c458d45aadf41bb74e168f66c8715b1c"
server_key_iv: "b680c30590dffde97f7c16b4bddd694b"
server_key_version: "encryptionkey@0"
I20260923 22:20:42.442307 21698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:42.458698 21698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:42.461118 21698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:42.461283 21698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:42.461583 21698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:42.461906 21698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:42.461959 21698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:42.462003 21698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:42.462034 21698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:42.467617 21698 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:33009
I20260923 22:20:42.467749 21826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:33009 every 8 connection(s)
I20260923 22:20:42.468053 21698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:20:42.468569 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 21698
I20260923 22:20:42.469219 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.468807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:54733)
0923 22:20:42.468978 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.469065 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
W20260923 22:20:42.469635 21827 heartbeater.cc:644] Failed to heartbeat to 127.4.207.62:38315 (0 consecutive failures): Network error: Failed to ping master at 127.4.207.62:38315: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
I20260923 22:20:42.469882 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.469812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:38687)
0923 22:20:42.469840 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.469865 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260923 22:20:42.470158 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:42.470092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:36125)
0923 22:20:42.470128 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:42.470147 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 22:20:42.470269 21827 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 22:20:43.273947 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.273703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:34369)
0923 22:20:43.273838 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:43.273878 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260923 22:20:43.338892 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.338701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:56357)
0923 22:20:43.338807 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:43.338842 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260923 22:20:43.470765 21833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.470570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:43105)
0923 22:20:43.470683 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:43.470717 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260923 22:20:43.773975 21834 raft_consensus.cc:482] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:43.774096 21834 raft_consensus.cc:504] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:43.774518 21834 leader_election.cc:288] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009), 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037)
Sep 23 22:20:43 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.2@KRBTEST.COM
Sep 23 22:20:43 dist-test-slave-7511 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202042, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.3@KRBTEST.COM
I20260923 22:20:43.785365 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.774788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:43911 (local address 127.4.207.3:33009)
0923 22:20:43.774945 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:43.774959 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:43.774995 (+ 36us) server_negotiation.cc:407] Connection header received
0923 22:20:43.775606 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:43.775609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:43.775666 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:43.775815 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:43.776605 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:43.777555 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:43.778411 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:43.778823 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:43.778938 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:43.781973 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:43.781976 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:43.781976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:43.781985 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:43.783759 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:43.784116 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:43.784118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:43.784118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:43.784205 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:43.784399 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:43.784400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:43.784400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:43.784684 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:43.784763 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:43.785117 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 22:20:43.785243 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260923 22:20:43.785387 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.774686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:39037 (local address 127.4.207.1:41587)
0923 22:20:43.774958 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:43.774968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:43.775017 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:43.775943 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:43.775944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:43.776256 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:43.776452 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:43.776458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:43.777553 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:43.777554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:43.778299 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:43.778301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:43.778898 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:43.778899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:43.779012 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:43.779777 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:43.779786 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:43.781393 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:43.783631 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:43.783637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:43.783638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:43.783917 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:43.784181 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:43.784181 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:43.784182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:43.784323 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:43.784759 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:43.784767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:43.784920 (+ 153us) client_negotiation.cc:769] Sending connection context
0923 22:20:43.785042 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 22:20:43.785243 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":23,"thread_start_us":46,"threads_started":1}
I20260923 22:20:43.785418 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.774832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:41587 (local address 127.4.207.2:39037)
0923 22:20:43.775032 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:43.775039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:43.775064 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:20:43.775602 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:43.775606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:43.775666 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:43.775815 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:43.776570 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:43.777450 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:43.778403 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:43.778792 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:43.778925 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:43.781514 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:43.781518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:43.781519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:43.781528 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:43.783559 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:43.783994 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:43.783996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:43.783996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:43.784089 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:43.784399 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:43.784400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:43.784400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:43.784685 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:43.784764 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:43.785127 (+ 363us) server_negotiation.cc:299] Negotiation successful
0923 22:20:43.785243 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":51,"threads_started":1}
I20260923 22:20:43.785454 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:43.774659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33009 (local address 127.4.207.1:43911)
0923 22:20:43.774794 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:43.774804 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:43.774909 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:43.775887 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:43.775890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:43.776258 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:43.776453 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:43.776459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:43.777651 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:43.777652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:43.778302 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:43.778304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:43.778919 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:43.778920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:43.779009 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:43.779843 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:43.779848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:43.781859 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:43.783829 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:43.783830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:43.783830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:43.784039 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:43.784274 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:43.784274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:43.784274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:43.784338 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:43.784759 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:43.784767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:43.784920 (+ 153us) client_negotiation.cc:769] Sending connection context
0923 22:20:43.785042 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 22:20:43.785324 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":117,"thread_start_us":65,"threads_started":1}
W20260923 22:20:43.786270 21450 leader_election.cc:341] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037): Not found: Tablet not found: 5f94b496273e4e84bf589d5310733841
W20260923 22:20:43.786839 21449 leader_election.cc:341] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009): Not found: Tablet not found: 5f94b496273e4e84bf589d5310733841
I20260923 22:20:43.786906 21449 leader_election.cc:302] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b6754039a6d4657bbca17891f115631; no voters: 42310e349a584898ae2f3aa4ea51f4c9, fd7d617f91a3413988462c7963e04116
I20260923 22:20:43.787045 21834 raft_consensus.cc:2726] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:20:44.274592 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:44.274466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:45541)
0923 22:20:44.274516 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:44.274549 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:20:44.339516 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:44.339295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:37425)
0923 22:20:44.339420 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:44.339476 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260923 22:20:44.471321 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:44.471149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:57681)
0923 22:20:44.471253 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:44.471286 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260923 22:20:45.275301 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:45.275064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:44201)
0923 22:20:45.275184 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:45.275240 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260923 22:20:45.340112 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:45.339929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:45307)
0923 22:20:45.340038 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:45.340072 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260923 22:20:45.471994 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:45.471754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:38867)
0923 22:20:45.471916 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:45.471953 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260923 22:20:45.938126 21844 raft_consensus.cc:482] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:45.938231 21844 raft_consensus.cc:504] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:45.938427 21844 leader_election.cc:288] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009), 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037)
W20260923 22:20:45.938856 21449 leader_election.cc:341] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009): Not found: Tablet not found: 5f94b496273e4e84bf589d5310733841
W20260923 22:20:45.938988 21450 leader_election.cc:341] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037): Not found: Tablet not found: 5f94b496273e4e84bf589d5310733841
I20260923 22:20:45.939028 21450 leader_election.cc:302] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b6754039a6d4657bbca17891f115631; no voters: 42310e349a584898ae2f3aa4ea51f4c9, fd7d617f91a3413988462c7963e04116
I20260923 22:20:45.939136 21844 raft_consensus.cc:2726] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:20:46.275982 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:46.275765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:49239)
0923 22:20:46.275899 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:46.275935 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":109,"threads_started":1}
I20260923 22:20:46.340981 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:46.340559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:45963)
0923 22:20:46.340690 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:46.340732 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260923 22:20:46.472676 21847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:46.472447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:41205)
0923 22:20:46.472577 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:46.472631 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260923 22:20:47.276695 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:47.276465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:33933)
0923 22:20:47.276611 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:47.276646 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260923 22:20:47.341732 21849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:47.341524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:51315)
0923 22:20:47.341658 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:47.341691 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":56,"threads_started":1}
I20260923 22:20:47.473295 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:47.473124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:54967)
0923 22:20:47.473232 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:47.473259 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260923 22:20:47.796607 21851 raft_consensus.cc:482] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:47.796700 21851 raft_consensus.cc:504] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:47.796865 21851 leader_election.cc:288] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009), 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037)
W20260923 22:20:47.797188 21449 leader_election.cc:341] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009): Not found: Tablet not found: 5f94b496273e4e84bf589d5310733841
W20260923 22:20:47.797262 21450 leader_election.cc:341] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037): Not found: Tablet not found: 5f94b496273e4e84bf589d5310733841
I20260923 22:20:47.797377 21450 leader_election.cc:302] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b6754039a6d4657bbca17891f115631; no voters: 42310e349a584898ae2f3aa4ea51f4c9, fd7d617f91a3413988462c7963e04116
I20260923 22:20:47.797456 21851 raft_consensus.cc:2726] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:20:48.277392 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:48.277169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:43913)
0923 22:20:48.277293 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:48.277336 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
I20260923 22:20:48.342394 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:48.342202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:33653)
0923 22:20:48.342320 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:48.342352 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260923 22:20:48.473961 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:48.473753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:50923)
0923 22:20:48.473873 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:48.473915 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260923 22:20:49.278206 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:49.277979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:56413)
0923 22:20:49.278118 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:49.278158 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260923 22:20:49.343080 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:49.342862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:37731)
0923 22:20:49.342997 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:49.343036 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260923 22:20:49.474696 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:49.474486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:57601)
0923 22:20:49.474619 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:49.474654 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260923 22:20:50.278914 21858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:50.278645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:60601)
0923 22:20:50.278791 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:50.278867 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260923 22:20:50.343840 21859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:50.343606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:36951)
0923 22:20:50.343762 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:50.343797 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260923 22:20:50.475536 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:50.475262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:45903)
0923 22:20:50.475407 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:50.475485 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260923 22:20:51.279742 21861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:51.279535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:60923)
0923 22:20:51.279652 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:51.279691 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260923 22:20:51.344658 21862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:51.344423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:46053)
0923 22:20:51.344580 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:51.344613 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260923 22:20:51.476222 21863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:51.475962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:39165)
0923 22:20:51.476132 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:51.476175 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260923 22:20:52.280520 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:52.280276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:38261)
0923 22:20:52.280432 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:52.280473 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260923 22:20:52.345363 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:52.345106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:47957)
0923 22:20:52.345259 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:52.345298 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260923 22:20:52.421463 21866 raft_consensus.cc:482] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:52.421571 21866 raft_consensus.cc:504] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd7d617f91a3413988462c7963e04116" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33009 } } peers { permanent_uuid: "42310e349a584898ae2f3aa4ea51f4c9" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 39037 } } peers { permanent_uuid: "7b6754039a6d4657bbca17891f115631" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45671 } }
I20260923 22:20:52.421789 21866 leader_election.cc:288] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009), 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037)
W20260923 22:20:52.422188 21449 leader_election.cc:341] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd7d617f91a3413988462c7963e04116 (127.4.207.3:33009): Not found: Tablet not found: 5f94b496273e4e84bf589d5310733841
W20260923 22:20:52.422266 21450 leader_election.cc:341] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42310e349a584898ae2f3aa4ea51f4c9 (127.4.207.2:39037): Not found: Tablet not found: 5f94b496273e4e84bf589d5310733841
I20260923 22:20:52.422340 21450 leader_election.cc:302] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b6754039a6d4657bbca17891f115631; no voters: 42310e349a584898ae2f3aa4ea51f4c9, fd7d617f91a3413988462c7963e04116
I20260923 22:20:52.422525 21866 raft_consensus.cc:2726] T 5f94b496273e4e84bf589d5310733841 P 7b6754039a6d4657bbca17891f115631 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 22:20:52.476945 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:52.476690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:40261)
0923 22:20:52.476855 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:52.476894 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260923 22:20:53.281260 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:53.281042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:44161)
0923 22:20:53.281174 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:53.281212 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
I20260923 22:20:53.346002 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:53.345795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:36395)
0923 22:20:53.345922 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:53.345959 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260923 22:20:53.477689 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:53.477447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:55199)
0923 22:20:53.477595 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:53.477640 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260923 22:20:54.281960 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:54.281737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:41175)
0923 22:20:54.281875 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:54.281911 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260923 22:20:54.346662 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:54.346469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:46941)
0923 22:20:54.346584 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:54.346619 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260923 22:20:54.478422 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:54.478219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:42785)
0923 22:20:54.478342 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:54.478379 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260923 22:20:55.282688 21874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.282432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:59815)
0923 22:20:55.282597 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.282638 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":71,"threads_started":1}
I20260923 22:20:55.347306 21875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.347082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:52597)
0923 22:20:55.347207 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.347244 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260923 22:20:55.479146 21876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:55.478908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:47243)
0923 22:20:55.479061 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:55.479100 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260923 22:20:56.283679 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.283166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:52399)
0923 22:20:56.283590 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.283630 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":109,"threads_started":1}
I20260923 22:20:56.348048 21878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.347806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:53381)
0923 22:20:56.347964 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.348000 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260923 22:20:56.479869 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:56.479656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.3:39955)
0923 22:20:56.479783 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:56.479823 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260923 22:20:57.284534 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:57.284317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.1:50479)
0923 22:20:57.284451 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:57.284485 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":59,"threads_started":1}
I20260923 22:20:57.348809 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:57.348537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:38315 (local address 127.4.207.2:43107)
0923 22:20:57.348714 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:57.348763 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.62:38315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":97,"threads_started":1}
I20260923 22:20:57.470965 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 21400
I20260923 22:20:57.475764 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 21565
I20260923 22:20:57.480057 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 21698
2026-09-23T22:20:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58190 ms)
[----------] 30 tests from SecurityITest (229895 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:20:57 dist-test-slave-7511 krb5kdc[21884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:20:57 dist-test-slave-7511 krb5kdc[21884](info): set up 2 sockets
Sep 23 22:20:57 dist-test-slave-7511 krb5kdc[21884](info): commencing operation
krb5kdc: starting...
W20260923 22:20:58.571131 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:20:58 dist-test-slave-7511 krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:20:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:20:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:20:58.690361 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:41545
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:41545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:58.768643 21897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:58.768850 21897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:58.768872 21897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:58.769912 21897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:58.769944 21897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:58.769958 21897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:58.769970 21897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:58.769981 21897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:58.771337 21897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:41545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:41545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:58.771584 21897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:58.771757 21897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:20:58.777022 21897 server_base.cc:1034] running on GCE node
W20260923 22:20:58.777004 21905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.776990 21902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.776998 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:58.777480 21897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:58.777787 21897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:58.778970 21897 hybrid_clock.cc:648] HybridClock initialized: now 1790202058778955 us; error 30 us; skew 500 ppm
Sep 23 22:20:58 dist-test-slave-7511 krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:58.781638 21897 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:20:58.782171 21897 webserver.cc:460] Webserver started at http://127.4.207.62:38421/ using document root <none> and password file <none>
I20260923 22:20:58.782403 21897 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:58.782474 21897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:58.782626 21897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:58.783787 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "bf5a63741c8c4af8994d80970c78b047"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-7511"
server_key: "32f74a654c181e712a89c0c2034dc307"
server_key_iv: "bc959469120f2f0f108d6eeb4c907ff7"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.784181 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "bf5a63741c8c4af8994d80970c78b047"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-7511"
server_key: "32f74a654c181e712a89c0c2034dc307"
server_key_iv: "bc959469120f2f0f108d6eeb4c907ff7"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.785547 21897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:20:58.786396 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.786643 21897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:58.786756 21897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "bf5a63741c8c4af8994d80970c78b047"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-7511"
server_key: "32f74a654c181e712a89c0c2034dc307"
server_key_iv: "bc959469120f2f0f108d6eeb4c907ff7"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.786857 21897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:58.806490 21897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:58.808959 21897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:58.809126 21897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:58.813047 21897 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:41545
I20260923 22:20:58.813102 21964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:41545 every 8 connection(s)
I20260923 22:20:58.813449 21897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:58.814293 21965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:58.815140 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 21897
I20260923 22:20:58.815260 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:20:58.815514 4924 external_mini_cluster.cc:1428] Setting key 18dd604f6632345b00a3eae82967e92d
I20260923 22:20:58.817646 21965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047: Bootstrap starting.
I20260923 22:20:58.818238 21965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:58.818523 21965 log.cc:824] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:58.819177 21965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047: No bootstrap required, opened a new log
I20260923 22:20:58.820956 21965 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5a63741c8c4af8994d80970c78b047" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41545 } }
I20260923 22:20:58.821115 21965 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:58.821161 21965 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5a63741c8c4af8994d80970c78b047, State: Initialized, Role: FOLLOWER
I20260923 22:20:58.821305 21965 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5a63741c8c4af8994d80970c78b047" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41545 } }
I20260923 22:20:58.821386 21965 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:58.821429 21965 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:58.821476 21965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:58.821969 21965 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5a63741c8c4af8994d80970c78b047" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41545 } }
I20260923 22:20:58.822093 21965 leader_election.cc:302] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5a63741c8c4af8994d80970c78b047; no voters:
Sep 23 22:20:58 dist-test-slave-7511 krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:58.822265 21965 leader_election.cc:288] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:20:58.822302 21970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:58.822495 21970 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [term 1 LEADER]: Becoming Leader. State: Replica: bf5a63741c8c4af8994d80970c78b047, State: Running, Role: LEADER
I20260923 22:20:58.822522 21965 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:58.822640 21970 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5a63741c8c4af8994d80970c78b047" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41545 } }
I20260923 22:20:58.823287 21972 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5a63741c8c4af8994d80970c78b047. Latest consensus state: current_term: 1 leader_uuid: "bf5a63741c8c4af8994d80970c78b047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5a63741c8c4af8994d80970c78b047" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41545 } } }
I20260923 22:20:58.823740 21972 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:58.823659 21971 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf5a63741c8c4af8994d80970c78b047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5a63741c8c4af8994d80970c78b047" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 41545 } } }
I20260923 22:20:58.824002 21984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:58.824033 21971 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047 [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:58.824573 21984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:58.826100 21984 catalog_manager.cc:1347] Generated new cluster ID: d599be34a290490d9444b48e6fb2ce6c
I20260923 22:20:58.826145 21984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:20:58.826390 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.816051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.4.207.62:41545)
0923 22:20:58.816213 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.816222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.816261 (+ 39us) server_negotiation.cc:407] Connection header received
0923 22:20:58.816668 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.816674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.816717 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.816829 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.817652 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.818697 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.819604 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.820021 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.820135 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.822709 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:58.822713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:58.822713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:58.822724 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:58.824811 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.825154 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.825157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.825157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.825250 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:58.825477 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:58.825478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:58.825478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:58.825797 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:58.825954 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:58.826050 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 22:20:58.826187 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260923 22:20:58.833096 21984 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:20:58.833593 21984 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:58.845085 21984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047: Generated new TSK 0
I20260923 22:20:58.845275 21984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:20:58.874241 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41545
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:58.955539 21991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:58.955768 21991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:58.955803 21991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:58.956954 21991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:58.957011 21991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:58.957053 21991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:20:58.958470 21991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:58.958720 21991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:58.958956 21991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:58.963804 21996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.963811 21999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:58.963811 21997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:58.964076 21991 server_base.cc:1034] running on GCE node
I20260923 22:20:58.964318 21991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:58.964634 21991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:58.965796 21991 hybrid_clock.cc:648] HybridClock initialized: now 1790202058965784 us; error 27 us; skew 500 ppm
Sep 23 22:20:58 dist-test-slave-7511 krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:58.968664 21991 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:20:58.969208 21991 webserver.cc:460] Webserver started at http://127.4.207.1:46057/ using document root <none> and password file <none>
I20260923 22:20:58.969413 21991 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:58.969462 21991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:58.969570 21991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:58.970686 21991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "86606da72c1e4fb1a2375ae5cd3af872"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-7511"
server_key: "510d255c6b20c32ef2769adf8bdc3208"
server_key_iv: "2966986018388be066cac71eeee28083"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.971038 21991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "86606da72c1e4fb1a2375ae5cd3af872"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-7511"
server_key: "510d255c6b20c32ef2769adf8bdc3208"
server_key_iv: "2966986018388be066cac71eeee28083"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.972560 21991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:20:58.973278 22006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.973544 21991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:58.973663 21991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "86606da72c1e4fb1a2375ae5cd3af872"
format_stamp: "Formatted at 2026-09-23 22:20:58 on dist-test-slave-7511"
server_key: "510d255c6b20c32ef2769adf8bdc3208"
server_key_iv: "2966986018388be066cac71eeee28083"
server_key_version: "encryptionkey@0"
I20260923 22:20:58.973757 21991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:58.983604 21991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:58.985818 21991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:58.986006 21991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:58.986256 21991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:58.986598 21991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:58.986747 21991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.986828 21991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:58.986860 21991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:58.992909 21991 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:42101
I20260923 22:20:58.992964 22119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:42101 every 8 connection(s)
I20260923 22:20:58.993268 21991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:58.999115 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 21991
I20260923 22:20:58.999240 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:20:58.999475 4924 external_mini_cluster.cc:1428] Setting key 7b270f76410ae904d85cb0f5a1f61822
Sep 23 22:20:58 dist-test-slave-7511 krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202058, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:59.003044 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.993954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:58857 (local address 127.4.207.62:41545)
0923 22:20:58.993995 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:58.993997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:58.994239 (+ 242us) server_negotiation.cc:407] Connection header received
0923 22:20:58.994697 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:58.994698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:58.994708 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:58.994761 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:58.995688 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.996176 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.997385 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.997608 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.997674 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.000077 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:59.000079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:59.000079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:59.000086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:59.001698 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:59.002093 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:59.002094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:59.002095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:59.002150 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:59.002398 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:59.002399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:59.002400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:59.002522 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:59.002597 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.002858 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.002943 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:20:59.003240 22122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:58.993948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41545 (local address 127.4.207.1:58857)
0923 22:20:58.994116 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:58.994151 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:58.994612 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:58.994814 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:58.994817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:58.995378 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:58.995603 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.995607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.996289 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.996290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.997252 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:58.997255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:58.997721 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:58.997722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:58.997831 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:58.998338 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:58.998346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:58.999951 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:59.001766 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:59.001773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:59.001776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:59.001995 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:59.002227 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:59.002228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:59.002228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:59.002332 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:59.002589 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:59.002595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:59.002732 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.002837 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.002993 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260923 22:20:59.003657 22120 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41545
I20260923 22:20:59.003746 22120 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:59.003940 22120 heartbeater.cc:511] Master 127.4.207.62:41545 requested a full tablet report, sending...
I20260923 22:20:59.004441 21929 ts_manager.cc:194] Registered new tserver with Master: 86606da72c1e4fb1a2375ae5cd3af872 (127.4.207.1:42101)
I20260923 22:20:59.005049 21929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:58857
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:20:59.024097 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41545
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:59.108268 22125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:59.108500 22125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:59.108541 22125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:59.109695 22125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:59.109767 22125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:59.109814 22125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:20:59.111213 22125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:59.111495 22125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:59.111743 22125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:59.117477 22133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.117532 22131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.117486 22130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:59.117689 22125 server_base.cc:1034] running on GCE node
I20260923 22:20:59.117897 22125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:59.118137 22125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:59.119293 22125 hybrid_clock.cc:648] HybridClock initialized: now 1790202059119284 us; error 39 us; skew 500 ppm
Sep 23 22:20:59 dist-test-slave-7511 krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202059, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:59.121829 22125 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:20:59.122246 22125 webserver.cc:460] Webserver started at http://127.4.207.2:43363/ using document root <none> and password file <none>
I20260923 22:20:59.122490 22125 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:59.122563 22125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:59.122700 22125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:59.123914 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "c2de750729d14dec9c29638047580616"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "0a2bb2735c34a8ee69fef54767b38ca5"
server_key_iv: "9e6e8cca57a3cd5808e91748571cce46"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.124209 22125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "c2de750729d14dec9c29638047580616"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "0a2bb2735c34a8ee69fef54767b38ca5"
server_key_iv: "9e6e8cca57a3cd5808e91748571cce46"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.125442 22125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:20:59.126164 22140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.126359 22125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:20:59.126433 22125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "c2de750729d14dec9c29638047580616"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "0a2bb2735c34a8ee69fef54767b38ca5"
server_key_iv: "9e6e8cca57a3cd5808e91748571cce46"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.126482 22125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:59.139359 22125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:59.141186 22125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:59.141294 22125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:59.141482 22125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:59.141717 22125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:59.141747 22125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.141768 22125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:59.141780 22125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.147542 22125 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:40861
I20260923 22:20:59.147608 22253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:40861 every 8 connection(s)
I20260923 22:20:59.147904 22125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:20:59.148788 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 22125
I20260923 22:20:59.148910 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:20:59.149147 4924 external_mini_cluster.cc:1428] Setting key 20019859761e82c443d4df6d4d99a68f
Sep 23 22:20:59 dist-test-slave-7511 krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202059, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
I20260923 22:20:59.158268 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.148654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:56025 (local address 127.4.207.62:41545)
0923 22:20:59.148733 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.148735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.148935 (+ 200us) server_negotiation.cc:407] Connection header received
0923 22:20:59.149617 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.149618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.149627 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.149667 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:59.150963 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.151552 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.152863 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.153065 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.153144 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.155345 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:59.155346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:59.155347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:59.155354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:59.156834 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:59.157251 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:59.157252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:59.157253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:59.157321 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:59.157587 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:59.157588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:59.157588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:59.157745 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:59.157819 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.158090 (+ 271us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.158175 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 22:20:59.158560 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.148594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41545 (local address 127.4.207.2:56025)
0923 22:20:59.148789 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.148837 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.149517 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.149735 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.149739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.150589 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:59.150861 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.150867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.151625 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.151627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.152793 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.152796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.153154 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.153155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.153292 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.153832 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:59.153840 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:59.155247 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:59.156943 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:59.156950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:59.156952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:59.157174 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:59.157389 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:59.157390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:59.157391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:59.157510 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:59.157811 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:59.157817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:59.157958 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.158067 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.158222 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260923 22:20:59.159049 22254 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41545
I20260923 22:20:59.159147 22254 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:59.159314 22254 heartbeater.cc:511] Master 127.4.207.62:41545 requested a full tablet report, sending...
I20260923 22:20:59.159775 21929 ts_manager.cc:194] Registered new tserver with Master: c2de750729d14dec9c29638047580616 (127.4.207.2:40861)
I20260923 22:20:59.160161 21929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:56025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:20:59.175473 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41545
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:44893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:20:59.255873 22259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:59.256074 22259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:59.256096 22259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:59.257133 22259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:59.257165 22259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:59.257190 22259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:20:59.258564 22259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:44893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:41545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:59.258754 22259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:59.258927 22259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:59.263841 22267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.263849 22265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.263849 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:59.264158 22259 server_base.cc:1034] running on GCE node
I20260923 22:20:59.264351 22259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:59.264554 22259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:59.265727 22259 hybrid_clock.cc:648] HybridClock initialized: now 1790202059265712 us; error 26 us; skew 500 ppm
Sep 23 22:20:59 dist-test-slave-7511 krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202059, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:20:59.268234 22259 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:20:59.268718 22259 webserver.cc:460] Webserver started at http://127.4.207.3:40577/ using document root <none> and password file <none>
I20260923 22:20:59.269012 22259 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:59.269088 22259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:59.269222 22259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:59.270298 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "fedfd05186584f41a1bcb114eb7da2d6"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "ca263d4fa06e159ff0baf124d3e7e3b9"
server_key_iv: "cf37a579aeacc8918627f60909a8e1e0"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.270658 22259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "fedfd05186584f41a1bcb114eb7da2d6"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "ca263d4fa06e159ff0baf124d3e7e3b9"
server_key_iv: "cf37a579aeacc8918627f60909a8e1e0"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.272109 22259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:20:59.272881 22274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.273068 22259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:20:59.273170 22259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "fedfd05186584f41a1bcb114eb7da2d6"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "ca263d4fa06e159ff0baf124d3e7e3b9"
server_key_iv: "cf37a579aeacc8918627f60909a8e1e0"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.273241 22259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:59.284662 22259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:59.286722 22259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:59.286890 22259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:59.287124 22259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:59.287544 22259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:59.287613 22259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.287657 22259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:59.287683 22259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.293805 22259 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:40017
I20260923 22:20:59.293859 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:40017 every 8 connection(s)
I20260923 22:20:59.294162 22259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:20:59 dist-test-slave-7511 krb5kdc[21884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202059, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:20:59.300488 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 22259
I20260923 22:20:59.300597 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:20:59.300840 4924 external_mini_cluster.cc:1428] Setting key e00c17658a443fb5da90db0ef9cdc993
I20260923 22:20:59.303969 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.294878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:39077 (local address 127.4.207.62:41545)
0923 22:20:59.294922 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.294924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.295159 (+ 235us) server_negotiation.cc:407] Connection header received
0923 22:20:59.295765 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.295766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.295779 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.295829 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:59.296788 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.297289 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.298405 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.298611 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.298685 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.301005 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:59.301006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:59.301007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:59.301013 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:59.302641 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:59.303044 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:59.303045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:59.303045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:59.303097 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:59.303360 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:59.303360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:59.303361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:59.303514 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:59.303576 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.303840 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.303905 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:20:59.304273 22390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.294861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:41545 (local address 127.4.207.3:39077)
0923 22:20:59.295037 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.295072 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.295653 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.295905 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.295909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.296487 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:59.296680 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.296685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.297381 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.297382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.298302 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.298305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.298691 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.298692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.298823 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.299305 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:20:59.299313 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:59.300910 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:59.302731 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:59.302738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:59.302739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:59.302977 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:59.303179 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:20:59.303180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:20:59.303180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:20:59.303296 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:20:59.303587 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:59.303593 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:59.303720 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.303816 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.303946 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260923 22:20:59.304843 22388 heartbeater.cc:347] Connected to a master server at 127.4.207.62:41545
I20260923 22:20:59.304965 22388 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:59.305155 22388 heartbeater.cc:511] Master 127.4.207.62:41545 requested a full tablet report, sending...
I20260923 22:20:59.305516 21929 ts_manager.cc:194] Registered new tserver with Master: fedfd05186584f41a1bcb114eb7da2d6 (127.4.207.3:40017)
I20260923 22:20:59.305871 21929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:39077
I20260923 22:20:59.312920 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:20:59.320221 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.314403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.4.207.62:41545)
0923 22:20:59.314456 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.314458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.314579 (+ 121us) server_negotiation.cc:407] Connection header received
0923 22:20:59.314627 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.314628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.314637 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.314685 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:59.315236 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.315805 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.316510 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.316704 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.316751 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.317881 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:59.317882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:59.317882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:20:59.317888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:59.319239 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:59.319599 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:59.319600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:59.319600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:59.319659 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:20:59.319810 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:20:59.319810 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:20:59.319811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:20:59.319937 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:59.320006 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.320097 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.320158 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 22:20:59.321705 21929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:20:59.323410 21929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:20:59.329823 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.325834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:40017 (local address 127.0.0.1:46246)
0923 22:20:59.326030 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.326039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.326093 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.326446 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.326449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.326454 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.326527 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.326528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.327430 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.327432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.328654 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.328657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.329511 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.329512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.329607 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.329622 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.329697 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.329745 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":6,"thread_start_us":59,"threads_started":1}
I20260923 22:20:59.329879 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.326045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.4.207.3:40017)
0923 22:20:59.326207 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.326213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.326223 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:20:59.326265 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.326267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.326282 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.326358 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.326611 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.327331 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.328745 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.329424 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.329503 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.329563 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.329729 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.329782 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260923 22:20:59.330129 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.325633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:40861 (local address 127.0.0.1:36714)
0923 22:20:59.325924 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.325957 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.326022 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.326449 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.326450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.326452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.326524 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.326526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.327948 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.327949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.328935 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.328937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.329843 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.329844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.329916 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.329920 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.329954 (+ 34us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.329999 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":3,"thread_start_us":53,"threads_started":1}
I20260923 22:20:59.330444 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.325764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.4.207.2:40861)
0923 22:20:59.326214 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.326218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.326227 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:20:59.326265 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.326267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.326282 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.326604 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.326682 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.327836 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.329017 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.329746 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.330201 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.330285 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.330313 (+ 28us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.330359 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":70,"threads_started":1}
I20260923 22:20:59.331138 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.325628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:42101 (local address 127.0.0.1:51200)
0923 22:20:59.325907 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.325947 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.326037 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.326516 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.326517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.326519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.326581 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.326583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.329035 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.329035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.329951 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.329955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.330897 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.330898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.330970 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.330974 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.331023 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.331065 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":59,"threads_started":1}
I20260923 22:20:59.331017 22322 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec9d956ccc44b6982565ff5ff7cc62e (DEFAULT_TABLE table=test-table [id=840c3dfc04d3415ea9d48dfb9b001f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:59.331127 22188 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec9d956ccc44b6982565ff5ff7cc62e (DEFAULT_TABLE table=test-table [id=840c3dfc04d3415ea9d48dfb9b001f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:59.331421 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.325723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.4.207.1:42101)
0923 22:20:59.325882 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.325887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.326123 (+ 236us) server_negotiation.cc:407] Connection header received
0923 22:20:59.326170 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.326172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.326189 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.326441 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.326658 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.327846 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.330043 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.330753 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.330817 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.330865 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.331291 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.331339 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260923 22:20:59.331418 22322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec9d956ccc44b6982565ff5ff7cc62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:59.331516 22188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec9d956ccc44b6982565ff5ff7cc62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:59.334096 22402 tablet_bootstrap.cc:492] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Bootstrap starting.
I20260923 22:20:59.334138 22403 tablet_bootstrap.cc:492] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Bootstrap starting.
I20260923 22:20:59.334754 22402 tablet_bootstrap.cc:654] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:59.334681 22054 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec9d956ccc44b6982565ff5ff7cc62e (DEFAULT_TABLE table=test-table [id=840c3dfc04d3415ea9d48dfb9b001f01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:20:59.334868 22403 tablet_bootstrap.cc:654] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:59.334996 22054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec9d956ccc44b6982565ff5ff7cc62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:59.335315 22402 log.cc:824] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:59.335474 22403 log.cc:824] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:59.336030 22402 tablet_bootstrap.cc:492] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: No bootstrap required, opened a new log
I20260923 22:20:59.336082 22402 ts_tablet_manager.cc:1397] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:20:59.337601 22402 raft_consensus.cc:348] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.337733 22402 raft_consensus.cc:374] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:59.337755 22402 raft_consensus.cc:729] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fedfd05186584f41a1bcb114eb7da2d6, State: Initialized, Role: FOLLOWER
I20260923 22:20:59.337874 22402 consensus_queue.cc:260] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.338047 22406 tablet_bootstrap.cc:492] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872: Bootstrap starting.
I20260923 22:20:59.338174 22402 ts_tablet_manager.cc:1428] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:59.338291 22388 heartbeater.cc:503] Master 127.4.207.62:41545 was elected leader, sending a full tablet report...
I20260923 22:20:59.338708 22406 tablet_bootstrap.cc:654] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:59.338879 22403 tablet_bootstrap.cc:492] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: No bootstrap required, opened a new log
I20260923 22:20:59.338953 22403 ts_tablet_manager.cc:1397] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260923 22:20:59.339346 22406 log.cc:824] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:59.340001 22406 tablet_bootstrap.cc:492] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872: No bootstrap required, opened a new log
I20260923 22:20:59.340045 22406 ts_tablet_manager.cc:1397] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:59.340494 22403 raft_consensus.cc:348] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.340642 22403 raft_consensus.cc:374] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:59.340664 22403 raft_consensus.cc:729] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2de750729d14dec9c29638047580616, State: Initialized, Role: FOLLOWER
I20260923 22:20:59.340744 22403 consensus_queue.cc:260] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.340922 22403 ts_tablet_manager.cc:1428] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:20:59.341022 22254 heartbeater.cc:503] Master 127.4.207.62:41545 was elected leader, sending a full tablet report...
I20260923 22:20:59.341791 22406 raft_consensus.cc:348] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.341987 22406 raft_consensus.cc:374] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:59.342038 22406 raft_consensus.cc:729] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86606da72c1e4fb1a2375ae5cd3af872, State: Initialized, Role: FOLLOWER
I20260923 22:20:59.342170 22406 consensus_queue.cc:260] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.342413 22406 ts_tablet_manager.cc:1428] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:20:59.342479 22120 heartbeater.cc:503] Master 127.4.207.62:41545 was elected leader, sending a full tablet report...
W20260923 22:20:59.398479 22255 tablet.cc:2369] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:20:59.494009 22121 tablet.cc:2369] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:20:59.544697 22389 tablet.cc:2369] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:20:59.552685 22409 raft_consensus.cc:482] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:20:59.552824 22409 raft_consensus.cc:504] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.553195 22409 leader_election.cc:288] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86606da72c1e4fb1a2375ae5cd3af872 (127.4.207.1:42101), fedfd05186584f41a1bcb114eb7da2d6 (127.4.207.3:40017)
I20260923 22:20:59.556232 22398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.553480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:60247 (local address 127.4.207.1:42101)
0923 22:20:59.553544 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.553546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.553554 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:20:59.553582 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.553583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.553593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.553662 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.553863 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.554427 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.555378 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.555821 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.555873 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.555882 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.556117 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.556158 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:20:59.556291 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.553503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:40017 (local address 127.4.207.2:35557)
0923 22:20:59.553677 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.553686 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.553735 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.553886 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.553886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.553888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.553983 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.553984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.554632 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.554633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.555398 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.555400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.556005 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.556006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.556062 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.556065 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.556140 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.556174 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":108,"threads_started":1}
I20260923 22:20:59.556270 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.553582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:35557 (local address 127.4.207.3:40017)
0923 22:20:59.553653 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.553655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.553776 (+ 121us) server_negotiation.cc:407] Connection header received
0923 22:20:59.553804 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.553805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.553819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.553869 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:20:59.554057 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.554551 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.555503 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.555901 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.555986 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.555997 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.556163 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.556205 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 22:20:59.556306 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.553380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:42101 (local address 127.4.207.2:60247)
0923 22:20:59.553446 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.553460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.553512 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.553679 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.553680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.553682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:20:59.553773 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.553775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.554508 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.554508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.555287 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.555289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.555928 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.555929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.556034 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.556037 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.556089 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.556145 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260923 22:20:59.556635 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec9d956ccc44b6982565ff5ff7cc62e" candidate_uuid: "c2de750729d14dec9c29638047580616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fedfd05186584f41a1bcb114eb7da2d6" is_pre_election: true
I20260923 22:20:59.556644 22074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec9d956ccc44b6982565ff5ff7cc62e" candidate_uuid: "c2de750729d14dec9c29638047580616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86606da72c1e4fb1a2375ae5cd3af872" is_pre_election: true
I20260923 22:20:59.556785 22342 raft_consensus.cc:2454] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2de750729d14dec9c29638047580616 in term 0.
I20260923 22:20:59.556785 22074 raft_consensus.cc:2454] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2de750729d14dec9c29638047580616 in term 0.
I20260923 22:20:59.557061 22144 leader_election.cc:302] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86606da72c1e4fb1a2375ae5cd3af872, c2de750729d14dec9c29638047580616; no voters:
I20260923 22:20:59.557186 22409 raft_consensus.cc:2781] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:20:59.557240 22409 raft_consensus.cc:482] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:20:59.557264 22409 raft_consensus.cc:3037] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:59.557971 22409 raft_consensus.cc:504] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.558096 22409 leader_election.cc:288] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [CANDIDATE]: Term 1 election: Requested vote from peers 86606da72c1e4fb1a2375ae5cd3af872 (127.4.207.1:42101), fedfd05186584f41a1bcb114eb7da2d6 (127.4.207.3:40017)
I20260923 22:20:59.558305 22074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec9d956ccc44b6982565ff5ff7cc62e" candidate_uuid: "c2de750729d14dec9c29638047580616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86606da72c1e4fb1a2375ae5cd3af872"
I20260923 22:20:59.558305 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec9d956ccc44b6982565ff5ff7cc62e" candidate_uuid: "c2de750729d14dec9c29638047580616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fedfd05186584f41a1bcb114eb7da2d6"
I20260923 22:20:59.558382 22342 raft_consensus.cc:3037] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:59.558383 22074 raft_consensus.cc:3037] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:59.559238 22074 raft_consensus.cc:2454] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2de750729d14dec9c29638047580616 in term 1.
I20260923 22:20:59.559238 22342 raft_consensus.cc:2454] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2de750729d14dec9c29638047580616 in term 1.
I20260923 22:20:59.559494 22144 leader_election.cc:302] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2de750729d14dec9c29638047580616, fedfd05186584f41a1bcb114eb7da2d6; no voters:
I20260923 22:20:59.559597 22409 raft_consensus.cc:2781] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:59.559744 22409 raft_consensus.cc:686] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 1 LEADER]: Becoming Leader. State: Replica: c2de750729d14dec9c29638047580616, State: Running, Role: LEADER
I20260923 22:20:59.559839 22409 consensus_queue.cc:237] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } }
I20260923 22:20:59.560660 21929 catalog_manager.cc:5495] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2de750729d14dec9c29638047580616 (127.4.207.2). New cstate: current_term: 1 leader_uuid: "c2de750729d14dec9c29638047580616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2de750729d14dec9c29638047580616" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 40861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 } health_report { overall_health: UNKNOWN } } }
I20260923 22:20:59.612418 22399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.610071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.4.207.2:40861)
0923 22:20:59.610135 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.610137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.610150 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:20:59.610224 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.610225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.610245 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.610309 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:20:59.610439 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.610887 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.611616 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.611795 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.611849 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.612106 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:20:59.612213 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:20:59.612287 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.612332 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.612351 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:20:59.614566 22074 raft_consensus.cc:1261] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [term 1 FOLLOWER]: Refusing update from remote peer c2de750729d14dec9c29638047580616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:59.614565 22342 raft_consensus.cc:1261] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 1 FOLLOWER]: Refusing update from remote peer c2de750729d14dec9c29638047580616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:20:59.615015 22409 consensus_queue.cc:1035] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fedfd05186584f41a1bcb114eb7da2d6" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:59.615216 22412 consensus_queue.cc:1035] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86606da72c1e4fb1a2375ae5cd3af872" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:20:59.617614 22415 mvcc.cc:205] Tried to move back new op lower bound from 7332667636177489920 to 7332667635957608448. Current Snapshot: MvccSnapshot[applied={T|T < 7332667636177489920}]
I20260923 22:20:59.617903 22417 mvcc.cc:205] Tried to move back new op lower bound from 7332667636177489920 to 7332667635957608448. Current Snapshot: MvccSnapshot[applied={T|T < 7332667636177489920}]
I20260923 22:20:59.620363 21929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:59.620469 21929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:20:59.621199 21929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf5a63741c8c4af8994d80970c78b047: Sending DeleteTablet for 3 replicas of tablet 9ec9d956ccc44b6982565ff5ff7cc62e
I20260923 22:20:59.621528 22188 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec9d956ccc44b6982565ff5ff7cc62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:59 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:36714
I20260923 22:20:59.621541 22054 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec9d956ccc44b6982565ff5ff7cc62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:59 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:51200
I20260923 22:20:59.621739 22425 tablet_replica.cc:330] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872: stopping tablet replica
I20260923 22:20:59.621814 22322 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec9d956ccc44b6982565ff5ff7cc62e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:20:59 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:46246
I20260923 22:20:59.621901 22426 tablet_replica.cc:330] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: stopping tablet replica
I20260923 22:20:59.621968 22425 raft_consensus.cc:2229] T 9ec9d956ccc44b6982565ff5ff7cc62e P 86606da72c1e4fb1a2375ae5cd3af872 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:59.622133 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 21991
I20260923 22:20:59.622823 22426 raft_consensus.cc:2229] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:20:59.622956 22426 raft_consensus.cc:2258] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:59.623097 22427 tablet_replica.cc:330] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: stopping tablet replica
I20260923 22:20:59.623183 22427 raft_consensus.cc:2229] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:20:59.623276 22427 raft_consensus.cc:2258] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:20:59.623791 22427 ts_tablet_manager.cc:1905] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:59.624109 22426 ts_tablet_manager.cc:1905] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:20:59.625840 22426 ts_tablet_manager.cc:1918] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:59.625891 22427 ts_tablet_manager.cc:1918] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:20:59.625959 22426 log.cc:1196] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/9ec9d956ccc44b6982565ff5ff7cc62e
I20260923 22:20:59.625990 22427 log.cc:1196] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/9ec9d956ccc44b6982565ff5ff7cc62e
I20260923 22:20:59.626271 22426 ts_tablet_manager.cc:1939] T 9ec9d956ccc44b6982565ff5ff7cc62e P c2de750729d14dec9c29638047580616: Deleting consensus metadata
I20260923 22:20:59.626459 22427 ts_tablet_manager.cc:1939] T 9ec9d956ccc44b6982565ff5ff7cc62e P fedfd05186584f41a1bcb114eb7da2d6: Deleting consensus metadata
I20260923 22:20:59.626685 21913 catalog_manager.cc:4841] TS c2de750729d14dec9c29638047580616 (127.4.207.2:40861): tablet 9ec9d956ccc44b6982565ff5ff7cc62e (table test-table [id=840c3dfc04d3415ea9d48dfb9b001f01]) successfully deleted
I20260923 22:20:59.626852 21916 catalog_manager.cc:4841] TS fedfd05186584f41a1bcb114eb7da2d6 (127.4.207.3:40017): tablet 9ec9d956ccc44b6982565ff5ff7cc62e (table test-table [id=840c3dfc04d3415ea9d48dfb9b001f01]) successfully deleted
W20260923 22:20:59.627756 21916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:42101 (error 108)
I20260923 22:20:59.628016 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 22125
I20260923 22:20:59.628656 22400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.628136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:42101 (local address 127.0.0.1:51212)
0923 22:20:59.628564 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.628604 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:42101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408}
W20260923 22:20:59.628808 21916 catalog_manager.cc:4584] TS 86606da72c1e4fb1a2375ae5cd3af872 (127.4.207.1:42101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ec9d956ccc44b6982565ff5ff7cc62e: Network error: Client connection negotiation failed: client connection to 127.4.207.1:42101: connect: Connection refused (error 111)
I20260923 22:20:59.632757 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 22259
I20260923 22:20:59.637157 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 21897
2026-09-23T22:20:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2157 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T22:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:20:59Z Disabled control of system clock
I20260923 22:20:59.664007 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39919
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:39919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:20:59.742192 22432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:59.742400 22432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:59.742420 22432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:59.743543 22432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:20:59.743580 22432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:59.743592 22432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:59.743604 22432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:20:59.743655 22432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:20:59.744894 22432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:39919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:59.745131 22432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:59.745406 22432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:59.750061 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.750072 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.750104 22437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:59.750239 22432 server_base.cc:1034] running on GCE node
I20260923 22:20:59.750519 22432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:59.750818 22432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:59.752009 22432 hybrid_clock.cc:648] HybridClock initialized: now 1790202059751986 us; error 34 us; skew 500 ppm
I20260923 22:20:59.753252 22432 webserver.cc:460] Webserver started at http://127.4.207.62:39439/ using document root <none> and password file <none>
I20260923 22:20:59.753501 22432 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:59.753567 22432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:59.753715 22432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:59.754920 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "9e7c1b216b9d47e4b7aff22971c61e7a"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "d7ba51beda4cbf773efc19c6901010b7"
server_key_iv: "85b6560d573fe4626bd37e1aea21b552"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.755345 22432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "9e7c1b216b9d47e4b7aff22971c61e7a"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "d7ba51beda4cbf773efc19c6901010b7"
server_key_iv: "85b6560d573fe4626bd37e1aea21b552"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.756821 22432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:59.757727 22446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.757932 22432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:59.758047 22432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "9e7c1b216b9d47e4b7aff22971c61e7a"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "d7ba51beda4cbf773efc19c6901010b7"
server_key_iv: "85b6560d573fe4626bd37e1aea21b552"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.758126 22432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:59.765985 22432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:59.766250 22432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:59.766405 22432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:59.770422 22432 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:39919
I20260923 22:20:59.770453 22498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:39919 every 8 connection(s)
I20260923 22:20:59.770812 22432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:20:59.771811 22499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:20:59.773945 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a: Bootstrap starting.
I20260923 22:20:59.774441 22499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a: Neither blocks nor log segments found. Creating new log.
I20260923 22:20:59.774674 22499 log.cc:824] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a: Log is configured to *not* fsync() on all Append() calls
I20260923 22:20:59.775349 22499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a: No bootstrap required, opened a new log
I20260923 22:20:59.776890 22499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c1b216b9d47e4b7aff22971c61e7a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39919 } }
I20260923 22:20:59.777050 22499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:20:59.777076 22499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7c1b216b9d47e4b7aff22971c61e7a, State: Initialized, Role: FOLLOWER
I20260923 22:20:59.777207 22499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c1b216b9d47e4b7aff22971c61e7a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39919 } }
I20260923 22:20:59.777287 22499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:20:59.777310 22499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:20:59.777338 22499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:20:59.777830 22499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c1b216b9d47e4b7aff22971c61e7a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39919 } }
I20260923 22:20:59.777912 22499 leader_election.cc:302] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e7c1b216b9d47e4b7aff22971c61e7a; no voters:
I20260923 22:20:59.778048 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 22432
I20260923 22:20:59.778097 22499 leader_election.cc:288] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:20:59.778190 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:20:59.778179 22502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:20:59.778352 22499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:20:59.778465 4924 external_mini_cluster.cc:1428] Setting key fd907b94f066955d14d633ecba3a3a9d
I20260923 22:20:59.778472 22502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [term 1 LEADER]: Becoming Leader. State: Replica: 9e7c1b216b9d47e4b7aff22971c61e7a, State: Running, Role: LEADER
I20260923 22:20:59.778573 22502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c1b216b9d47e4b7aff22971c61e7a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39919 } }
I20260923 22:20:59.778988 22504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e7c1b216b9d47e4b7aff22971c61e7a. Latest consensus state: current_term: 1 leader_uuid: "9e7c1b216b9d47e4b7aff22971c61e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c1b216b9d47e4b7aff22971c61e7a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39919 } } }
I20260923 22:20:59.779072 22503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e7c1b216b9d47e4b7aff22971c61e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c1b216b9d47e4b7aff22971c61e7a" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39919 } } }
I20260923 22:20:59.779078 22504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:59.779172 22503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a [sys.catalog]: This master's current role is: LEADER
I20260923 22:20:59.779423 22513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:20:59.780000 22513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:20:59.781711 22513 catalog_manager.cc:1347] Generated new cluster ID: 0da553431a3d43bd88cddc0423b74671
I20260923 22:20:59.781772 22513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:20:59.784865 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.779146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.4.207.62:39919)
0923 22:20:59.779801 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.779810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.779842 (+ 32us) server_negotiation.cc:407] Connection header received
0923 22:20:59.779972 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.779979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.780050 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.780341 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:59.780438 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.781530 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.782634 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.783248 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.783384 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.784131 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:59.784135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:59.784135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:59.784354 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:59.784395 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:59.784409 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:59.784492 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.784561 (+ 69us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.784641 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":85,"threads_started":1}
I20260923 22:20:59.788723 22513 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:20:59.789392 22513 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:20:59.793556 22513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a: Generated new TSK 0
I20260923 22:20:59.793752 22513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:20:59.801278 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39919
--builtin_ntp_servers=127.4.207.20:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:20:59.882247 22523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:20:59.882480 22523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:20:59.882513 22523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:20:59.883677 22523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:20:59.883733 22523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:20:59.883770 22523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:20:59.885100 22523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:20:59.885363 22523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:20:59.885660 22523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:20:59.891502 22529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.891484 22531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:20:59.891487 22528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:20:59.891749 22523 server_base.cc:1034] running on GCE node
I20260923 22:20:59.891978 22523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:20:59.892235 22523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:20:59.893409 22523 hybrid_clock.cc:648] HybridClock initialized: now 1790202059893386 us; error 38 us; skew 500 ppm
I20260923 22:20:59.894924 22523 webserver.cc:460] Webserver started at http://127.4.207.1:42473/ using document root <none> and password file <none>
I20260923 22:20:59.895203 22523 fs_manager.cc:359] Metadata directory not provided
I20260923 22:20:59.895273 22523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:20:59.895407 22523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:20:59.896519 22523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "6435f9c0cda84df3a8db9232a5565787"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "effcaf8d9f86ec8f905f4aa8fcacd5b6"
server_key_iv: "f8786394c03c91e3d4920e67b1bfdea3"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.896873 22523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "6435f9c0cda84df3a8db9232a5565787"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "effcaf8d9f86ec8f905f4aa8fcacd5b6"
server_key_iv: "f8786394c03c91e3d4920e67b1bfdea3"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.898233 22523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:20:59.899102 22537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.899259 22523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:20:59.899358 22523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "6435f9c0cda84df3a8db9232a5565787"
format_stamp: "Formatted at 2026-09-23 22:20:59 on dist-test-slave-7511"
server_key: "effcaf8d9f86ec8f905f4aa8fcacd5b6"
server_key_iv: "f8786394c03c91e3d4920e67b1bfdea3"
server_key_version: "encryptionkey@0"
I20260923 22:20:59.899483 22523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:20:59.915038 22523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:20:59.915375 22523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:20:59.915560 22523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:20:59.915795 22523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:20:59.916131 22523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:20:59.916190 22523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.916226 22523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:20:59.916251 22523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:20:59.921802 22523 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:40049
I20260923 22:20:59.921904 22650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:40049 every 8 connection(s)
I20260923 22:20:59.922220 22523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:20:59.926293 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 22523
I20260923 22:20:59.926422 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:20:59.926601 4924 external_mini_cluster.cc:1428] Setting key c5d685a7b5acc6a5ba756082d686ff9c
I20260923 22:20:59.927556 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.922951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:45193 (local address 127.4.207.62:39919)
0923 22:20:59.923037 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 22:20:59.923038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:20:59.923266 (+ 228us) server_negotiation.cc:407] Connection header received
0923 22:20:59.923812 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:20:59.923814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:20:59.923827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:20:59.923880 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:20:59.924552 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.925052 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.926064 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.926281 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.926358 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.927257 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:20:59.927258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:20:59.927259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:20:59.927265 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:20:59.927279 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:20:59.927290 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:20:59.927351 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:20:59.927430 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 22:20:59.927480 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:20:59.927572 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39919
--builtin_ntp_servers=127.4.207.20:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 22:20:59.927807 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:20:59.922947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39919 (local address 127.4.207.1:45193)
0923 22:20:59.923148 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 22:20:59.923181 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:20:59.923706 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:20:59.923948 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:20:59.923952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:20:59.923963 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:20:59.924450 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.924454 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.925155 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.925156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.925969 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:20:59.925971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:20:59.926365 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:20:59.926366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:20:59.926535 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:20:59.927100 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:20:59.927101 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:20:59.927128 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:20:59.927152 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:20:59.927351 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:20:59.927358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:20:59.927361 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:20:59.927451 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 22:20:59.927560 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260923 22:20:59.928419 22651 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39919
I20260923 22:20:59.928516 22651 heartbeater.cc:464] Registering TS with master...
I20260923 22:20:59.928735 22651 heartbeater.cc:511] Master 127.4.207.62:39919 requested a full tablet report, sending...
I20260923 22:20:59.929303 22463 ts_manager.cc:194] Registered new tserver with Master: 6435f9c0cda84df3a8db9232a5565787 (127.4.207.1:40049)
I20260923 22:20:59.929891 22463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.1:45193
W20260923 22:21:00.008742 22654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:00.008942 22654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:00.008963 22654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:00.010149 22654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:00.010191 22654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:00.010215 22654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:21:00.011576 22654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:00.011813 22654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:00.012110 22654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:00.017082 22659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:00.017053 22662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:00.017066 22660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:00.017335 22654 server_base.cc:1034] running on GCE node
I20260923 22:21:00.017565 22654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:00.017861 22654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:00.019032 22654 hybrid_clock.cc:648] HybridClock initialized: now 1790202060018994 us; error 53 us; skew 500 ppm
I20260923 22:21:00.020256 22654 webserver.cc:460] Webserver started at http://127.4.207.2:35473/ using document root <none> and password file <none>
I20260923 22:21:00.020494 22654 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:00.020557 22654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:00.020689 22654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:00.021667 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "da1fe9ea0a62445291b0fca08d10578f"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-7511"
server_key: "f733b546a5cab70c423dd2148bb6de10"
server_key_iv: "cc14914110b92b61a530f49c68cff9f6"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.021994 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "da1fe9ea0a62445291b0fca08d10578f"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-7511"
server_key: "f733b546a5cab70c423dd2148bb6de10"
server_key_iv: "cc14914110b92b61a530f49c68cff9f6"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.023204 22654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:00.023968 22668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.024120 22654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.024214 22654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "da1fe9ea0a62445291b0fca08d10578f"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-7511"
server_key: "f733b546a5cab70c423dd2148bb6de10"
server_key_iv: "cc14914110b92b61a530f49c68cff9f6"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.024294 22654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:00.045655 22654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:00.045966 22654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:00.046113 22654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:00.046339 22654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:00.046674 22654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:00.046729 22654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.046768 22654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:00.046795 22654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.052615 22654 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:44133
I20260923 22:21:00.052661 22781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:44133 every 8 connection(s)
I20260923 22:21:00.052974 22654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:21:00.058100 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.053686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:44441 (local address 127.4.207.62:39919)
0923 22:21:00.053768 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.053770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.053983 (+ 213us) server_negotiation.cc:407] Connection header received
0923 22:21:00.054494 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.054496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.054511 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.054553 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:00.055219 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.055708 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.056632 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.056807 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.056875 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.057745 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:00.057746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:00.057747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:00.057752 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:00.057766 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:00.057774 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:00.057841 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.057991 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.058022 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 22:21:00.058297 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.053656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39919 (local address 127.4.207.2:44441)
0923 22:21:00.053856 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.053889 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:00.054385 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:00.054629 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:00.054634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:00.054646 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:00.055114 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.055118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.055796 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.055797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.056534 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.056536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.056891 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.056892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.057014 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.057491 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:00.057491 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:00.057519 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:00.057543 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:00.057859 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:00.057867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:00.057875 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 22:21:00.057981 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 22:21:00.058085 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260923 22:21:00.058879 22782 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39919
I20260923 22:21:00.058997 22782 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:00.059212 22782 heartbeater.cc:511] Master 127.4.207.62:39919 requested a full tablet report, sending...
I20260923 22:21:00.059641 22463 ts_manager.cc:194] Registered new tserver with Master: da1fe9ea0a62445291b0fca08d10578f (127.4.207.2:44133)
I20260923 22:21:00.060173 22463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.2:44441
I20260923 22:21:00.063149 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 22654
I20260923 22:21:00.063263 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:21:00.063524 4924 external_mini_cluster.cc:1428] Setting key dd199f6c8fe09d266817f83ea19cf43a
I20260923 22:21:00.064453 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39919
--builtin_ntp_servers=127.4.207.20:42241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:00.144368 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:00.144603 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:00.144634 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:00.145744 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:00.145829 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:00.145871 22785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:21:00.147300 22785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:00.147603 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:00.147909 22785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:00.152788 22793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:00.152877 22790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:00.152804 22791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:00.152951 22785 server_base.cc:1034] running on GCE node
I20260923 22:21:00.153156 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:00.153394 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:00.154563 22785 hybrid_clock.cc:648] HybridClock initialized: now 1790202060154554 us; error 34 us; skew 500 ppm
I20260923 22:21:00.155997 22785 webserver.cc:460] Webserver started at http://127.4.207.3:41175/ using document root <none> and password file <none>
I20260923 22:21:00.156260 22785 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:00.156311 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:00.156493 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:00.157568 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "5405a7d5fb5949878ce88530c244c15c"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-7511"
server_key: "233aa53c7c926cf7cb24d8806d483905"
server_key_iv: "ed71e2c3df878775ea94ac676f9d20c1"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.157994 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "5405a7d5fb5949878ce88530c244c15c"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-7511"
server_key: "233aa53c7c926cf7cb24d8806d483905"
server_key_iv: "ed71e2c3df878775ea94ac676f9d20c1"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.159559 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260923 22:21:00.160386 22799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.160564 22785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:00.160665 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "5405a7d5fb5949878ce88530c244c15c"
format_stamp: "Formatted at 2026-09-23 22:21:00 on dist-test-slave-7511"
server_key: "233aa53c7c926cf7cb24d8806d483905"
server_key_iv: "ed71e2c3df878775ea94ac676f9d20c1"
server_key_version: "encryptionkey@0"
I20260923 22:21:00.160755 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:00.174626 22785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:00.174969 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:00.175132 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:00.175401 22785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:00.175772 22785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:00.175832 22785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.175879 22785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:00.175931 22785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:00.182384 22785 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:42059
I20260923 22:21:00.182452 22912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:42059 every 8 connection(s)
I20260923 22:21:00.182802 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:21:00.188047 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.183510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:37889 (local address 127.4.207.62:39919)
0923 22:21:00.183575 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.183576 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.183855 (+ 279us) server_negotiation.cc:407] Connection header received
0923 22:21:00.184326 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.184327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.184336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.184387 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:00.185055 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.185530 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.186556 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.186744 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.186802 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.187790 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:00.187791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:00.187791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:00.187795 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:00.187808 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:00.187812 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:00.187863 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.187966 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.187989 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:21:00.188331 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.183536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39919 (local address 127.4.207.3:37889)
0923 22:21:00.183717 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.183764 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:00.184244 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:00.184452 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:00.184456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:00.184463 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:00.184971 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.184975 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.185624 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.185626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.186461 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.186463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.186837 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.186838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.187005 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.187619 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:00.187620 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:00.187651 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:00.187695 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:00.187879 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:00.187883 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:00.187885 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:21:00.187977 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 22:21:00.188062 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260923 22:21:00.189028 22913 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39919
I20260923 22:21:00.189124 22913 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:00.189334 22913 heartbeater.cc:511] Master 127.4.207.62:39919 requested a full tablet report, sending...
I20260923 22:21:00.189674 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 22785
I20260923 22:21:00.189815 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:21:00.189832 22463 ts_manager.cc:194] Registered new tserver with Master: 5405a7d5fb5949878ce88530c244c15c (127.4.207.3:42059)
I20260923 22:21:00.190040 4924 external_mini_cluster.cc:1428] Setting key 09108f1656b846dde10ef2aa4762132f
I20260923 22:21:00.190315 22463 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.3:37889
I20260923 22:21:00.191082 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:21:00.195254 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.192501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34730 (local address 127.4.207.62:39919)
0923 22:21:00.192558 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.192560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.192646 (+ 86us) server_negotiation.cc:407] Connection header received
0923 22:21:00.192691 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.192692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.192701 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.192748 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:00.192873 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.193407 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.194111 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.194288 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.194350 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.195024 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:00.195025 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:00.195025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:00.195029 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:00.195041 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:00.195045 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:00.195083 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.195120 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.195138 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 22:21:00.196861 22463 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:21:00.198348 22463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:21:00.204792 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.200714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.4.207.1:40049)
0923 22:21:00.200856 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.200861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.201052 (+ 191us) server_negotiation.cc:407] Connection header received
0923 22:21:00.201094 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.201096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.201109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.201172 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:00.201464 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.202599 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.203945 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.204419 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.204509 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.204594 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.204678 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.204715 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260923 22:21:00.204888 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.200389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:44133 (local address 127.0.0.1:49484)
0923 22:21:00.200563 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.200606 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:00.200671 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:00.201095 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:00.201098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:00.201100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:00.201165 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.201167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.202518 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.202519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.203841 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.203843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.204656 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.204657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.204716 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.204721 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:21:00.204749 (+ 28us) client_negotiation.cc:240] Negotiation successful
0923 22:21:00.204786 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":29,"thread_start_us":81,"threads_started":1}
I20260923 22:21:00.204789 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.200609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:40049 (local address 127.0.0.1:36242)
0923 22:21:00.200926 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.200936 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:00.200993 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:00.201349 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:00.201349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:00.201351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:00.201389 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.201390 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.202648 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.202649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.203841 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.203843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.204514 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.204515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.204583 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.204595 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 22:21:00.204665 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:21:00.204715 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":33,"thread_start_us":77,"threads_started":1}
I20260923 22:21:00.204996 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.200502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.4.207.3:42059)
0923 22:21:00.200760 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.200766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.200940 (+ 174us) server_negotiation.cc:407] Connection header received
0923 22:21:00.201007 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.201009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.201023 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.201093 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:00.201277 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.202148 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.203860 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.204559 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.204628 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.204706 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.204860 (+ 154us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.204908 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260923 22:21:00.205209 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.200389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:42059 (local address 127.0.0.1:41046)
0923 22:21:00.200883 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.200892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:00.200943 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:00.201153 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:00.201154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:00.201155 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:00.201184 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.201185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.202237 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.202238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.203768 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.203771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.204733 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.204734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.204781 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.204783 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:21:00.205084 (+ 301us) client_negotiation.cc:240] Negotiation successful
0923 22:21:00.205134 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":175,"thread_start_us":65,"threads_started":1}
I20260923 22:21:00.205716 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.200758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.4.207.2:44133)
0923 22:21:00.200894 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.200898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.200910 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:21:00.200955 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.200957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.200973 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.201092 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:00.201242 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.202392 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.203930 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.204564 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.204830 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.205539 (+ 709us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.205564 (+ 25us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.205619 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260923 22:21:00.205659 22585 tablet_service.cc:1467] Processing CreateTablet for tablet ce768aef99814340bfcccdfd6608c870 (DEFAULT_TABLE table=test-table [id=fd3b4e111fcf4a49be2273e0c7ee924f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:00.205709 22847 tablet_service.cc:1467] Processing CreateTablet for tablet ce768aef99814340bfcccdfd6608c870 (DEFAULT_TABLE table=test-table [id=fd3b4e111fcf4a49be2273e0c7ee924f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:00.205991 22585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce768aef99814340bfcccdfd6608c870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:00.205991 22847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce768aef99814340bfcccdfd6608c870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:00.206276 22716 tablet_service.cc:1467] Processing CreateTablet for tablet ce768aef99814340bfcccdfd6608c870 (DEFAULT_TABLE table=test-table [id=fd3b4e111fcf4a49be2273e0c7ee924f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:00.206574 22716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce768aef99814340bfcccdfd6608c870. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:00.209244 22927 tablet_bootstrap.cc:492] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: Bootstrap starting.
I20260923 22:21:00.209569 22928 tablet_bootstrap.cc:492] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787: Bootstrap starting.
I20260923 22:21:00.209864 22927 tablet_bootstrap.cc:654] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:00.210148 22928 tablet_bootstrap.cc:654] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:00.210160 22927 log.cc:824] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:00.210419 22928 log.cc:824] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:00.210706 22929 tablet_bootstrap.cc:492] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: Bootstrap starting.
I20260923 22:21:00.210876 22927 tablet_bootstrap.cc:492] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: No bootstrap required, opened a new log
I20260923 22:21:00.210969 22927 ts_tablet_manager.cc:1397] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:00.211143 22928 tablet_bootstrap.cc:492] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787: No bootstrap required, opened a new log
I20260923 22:21:00.211205 22928 ts_tablet_manager.cc:1397] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:00.211362 22929 tablet_bootstrap.cc:654] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:00.211683 22929 log.cc:824] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:00.212352 22929 tablet_bootstrap.cc:492] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: No bootstrap required, opened a new log
I20260923 22:21:00.212428 22929 ts_tablet_manager.cc:1397] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:00.213843 22927 raft_consensus.cc:348] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.213851 22928 raft_consensus.cc:348] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.214022 22928 raft_consensus.cc:374] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:00.214022 22927 raft_consensus.cc:374] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:00.214059 22928 raft_consensus.cc:729] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6435f9c0cda84df3a8db9232a5565787, State: Initialized, Role: FOLLOWER
I20260923 22:21:00.214059 22927 raft_consensus.cc:729] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1fe9ea0a62445291b0fca08d10578f, State: Initialized, Role: FOLLOWER
I20260923 22:21:00.214200 22927 consensus_queue.cc:260] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.214200 22928 consensus_queue.cc:260] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.214460 22927 ts_tablet_manager.cc:1428] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260923 22:21:00.214460 22928 ts_tablet_manager.cc:1428] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:00.214402 22929 raft_consensus.cc:348] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.214560 22929 raft_consensus.cc:374] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:00.214581 22782 heartbeater.cc:503] Master 127.4.207.62:39919 was elected leader, sending a full tablet report...
I20260923 22:21:00.214581 22651 heartbeater.cc:503] Master 127.4.207.62:39919 was elected leader, sending a full tablet report...
I20260923 22:21:00.214603 22929 raft_consensus.cc:729] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5405a7d5fb5949878ce88530c244c15c, State: Initialized, Role: FOLLOWER
I20260923 22:21:00.214722 22929 consensus_queue.cc:260] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.214982 22929 ts_tablet_manager.cc:1428] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:00.215013 22913 heartbeater.cc:503] Master 127.4.207.62:39919 was elected leader, sending a full tablet report...
I20260923 22:21:00.238603 22934 raft_consensus.cc:482] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:21:00.238786 22934 raft_consensus.cc:504] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.239177 22934 leader_election.cc:288] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5405a7d5fb5949878ce88530c244c15c (127.4.207.3:42059), da1fe9ea0a62445291b0fca08d10578f (127.4.207.2:44133)
I20260923 22:21:00.242290 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.239502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:42059 (local address 127.4.207.1:45199)
0923 22:21:00.239562 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.239575 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:00.239627 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:00.239824 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:00.239825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:00.239826 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:00.239902 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.239905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.240616 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.240618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.241438 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.241440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.242049 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.242050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.242117 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.242120 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:00.242170 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:21:00.242203 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 22:21:00.242316 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.239601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:45199 (local address 127.4.207.3:42059)
0923 22:21:00.239681 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.239683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.239695 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:21:00.239732 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.239733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.239746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.239811 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:00.239979 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.240537 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.241537 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.241967 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.242026 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.242035 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.242197 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.242242 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 22:21:00.242388 22926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.239660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:40097 (local address 127.4.207.2:44133)
0923 22:21:00.239707 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.239711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.239806 (+ 95us) server_negotiation.cc:407] Connection header received
0923 22:21:00.239859 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.239860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.239870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.239928 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:00.240141 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.240608 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.241650 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.241982 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.242076 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.242085 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.242276 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.242320 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:21:00.242429 22936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.239515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:44133 (local address 127.4.207.1:40097)
0923 22:21:00.239720 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.239730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:00.239797 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:00.239974 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:00.239975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:00.239976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:00.240065 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.240067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.240707 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.240708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.241561 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:00.241564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.242107 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.242108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:00.242197 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.242202 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:21:00.242262 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 22:21:00.242299 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":14,"thread_start_us":134,"threads_started":1}
I20260923 22:21:00.242692 22867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce768aef99814340bfcccdfd6608c870" candidate_uuid: "6435f9c0cda84df3a8db9232a5565787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5405a7d5fb5949878ce88530c244c15c" is_pre_election: true
I20260923 22:21:00.242875 22867 raft_consensus.cc:2454] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6435f9c0cda84df3a8db9232a5565787 in term 0.
I20260923 22:21:00.242879 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce768aef99814340bfcccdfd6608c870" candidate_uuid: "6435f9c0cda84df3a8db9232a5565787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1fe9ea0a62445291b0fca08d10578f" is_pre_election: true
I20260923 22:21:00.243017 22736 raft_consensus.cc:2454] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6435f9c0cda84df3a8db9232a5565787 in term 0.
I20260923 22:21:00.243131 22540 leader_election.cc:302] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5405a7d5fb5949878ce88530c244c15c, 6435f9c0cda84df3a8db9232a5565787; no voters:
I20260923 22:21:00.243243 22934 raft_consensus.cc:2781] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:21:00.243285 22934 raft_consensus.cc:482] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:21:00.243310 22934 raft_consensus.cc:3037] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:00.244074 22934 raft_consensus.cc:504] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.244217 22934 leader_election.cc:288] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [CANDIDATE]: Term 1 election: Requested vote from peers 5405a7d5fb5949878ce88530c244c15c (127.4.207.3:42059), da1fe9ea0a62445291b0fca08d10578f (127.4.207.2:44133)
I20260923 22:21:00.244339 22867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce768aef99814340bfcccdfd6608c870" candidate_uuid: "6435f9c0cda84df3a8db9232a5565787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5405a7d5fb5949878ce88530c244c15c"
I20260923 22:21:00.244436 22867 raft_consensus.cc:3037] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:00.244417 22736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce768aef99814340bfcccdfd6608c870" candidate_uuid: "6435f9c0cda84df3a8db9232a5565787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da1fe9ea0a62445291b0fca08d10578f"
I20260923 22:21:00.244524 22736 raft_consensus.cc:3037] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:00.245208 22867 raft_consensus.cc:2454] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6435f9c0cda84df3a8db9232a5565787 in term 1.
I20260923 22:21:00.245307 22736 raft_consensus.cc:2454] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6435f9c0cda84df3a8db9232a5565787 in term 1.
I20260923 22:21:00.245390 22540 leader_election.cc:302] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5405a7d5fb5949878ce88530c244c15c, 6435f9c0cda84df3a8db9232a5565787; no voters:
I20260923 22:21:00.245503 22934 raft_consensus.cc:2781] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:00.245659 22934 raft_consensus.cc:686] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 1 LEADER]: Becoming Leader. State: Replica: 6435f9c0cda84df3a8db9232a5565787, State: Running, Role: LEADER
I20260923 22:21:00.245775 22934 consensus_queue.cc:237] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } }
I20260923 22:21:00.246611 22461 catalog_manager.cc:5495] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6435f9c0cda84df3a8db9232a5565787 (127.4.207.1). New cstate: current_term: 1 leader_uuid: "6435f9c0cda84df3a8db9232a5565787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6435f9c0cda84df3a8db9232a5565787" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 40049 } health_report { overall_health: HEALTHY } } }
I20260923 22:21:00.262481 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.260047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.4.207.1:40049)
0923 22:21:00.260109 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:00.260112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:00.260121 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:21:00.260194 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:00.260194 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:00.260220 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:00.260291 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:00.260432 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.260954 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.261703 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:00.261898 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:00.261972 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:00.262140 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:00.262243 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:00.262336 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:00.262354 (+ 18us) server_negotiation.cc:299] Negotiation successful
0923 22:21:00.262373 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:21:00.264524 22736 raft_consensus.cc:1261] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 1 FOLLOWER]: Refusing update from remote peer 6435f9c0cda84df3a8db9232a5565787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:00.264524 22867 raft_consensus.cc:1261] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 1 FOLLOWER]: Refusing update from remote peer 6435f9c0cda84df3a8db9232a5565787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:00.264940 22934 consensus_queue.cc:1035] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5405a7d5fb5949878ce88530c244c15c" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 42059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:00.265115 22937 consensus_queue.cc:1035] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da1fe9ea0a62445291b0fca08d10578f" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 44133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:00.267725 22941 mvcc.cc:205] Tried to move back new op lower bound from 7332667638840471552 to 7332667638767316992. Current Snapshot: MvccSnapshot[applied={T|T < 7332667638840471552}]
I20260923 22:21:00.268224 22942 mvcc.cc:205] Tried to move back new op lower bound from 7332667638840471552 to 7332667638767316992. Current Snapshot: MvccSnapshot[applied={T|T < 7332667638840471552}]
I20260923 22:21:00.269634 22461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:00.269738 22461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:00.270594 22461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e7c1b216b9d47e4b7aff22971c61e7a: Sending DeleteTablet for 3 replicas of tablet ce768aef99814340bfcccdfd6608c870
I20260923 22:21:00.270968 22847 tablet_service.cc:1514] Processing DeleteTablet for tablet ce768aef99814340bfcccdfd6608c870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:00 UTC) from {username='slave'} at 127.0.0.1:41046
I20260923 22:21:00.270980 22716 tablet_service.cc:1514] Processing DeleteTablet for tablet ce768aef99814340bfcccdfd6608c870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:00 UTC) from {username='slave'} at 127.0.0.1:49484
I20260923 22:21:00.270987 22585 tablet_service.cc:1514] Processing DeleteTablet for tablet ce768aef99814340bfcccdfd6608c870 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:00 UTC) from {username='slave'} at 127.0.0.1:36242
I20260923 22:21:00.271199 22951 tablet_replica.cc:330] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787: stopping tablet replica
I20260923 22:21:00.271289 22951 raft_consensus.cc:2229] T ce768aef99814340bfcccdfd6608c870 P 6435f9c0cda84df3a8db9232a5565787 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:21:00.271589 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 22523
I20260923 22:21:00.272277 22952 tablet_replica.cc:330] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: stopping tablet replica
I20260923 22:21:00.272372 22952 raft_consensus.cc:2229] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:00.272477 22952 raft_consensus.cc:2258] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:00.272650 22950 tablet_replica.cc:330] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: stopping tablet replica
I20260923 22:21:00.272742 22950 raft_consensus.cc:2229] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:00.272835 22950 raft_consensus.cc:2258] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:00.273243 22952 ts_tablet_manager.cc:1905] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:00.274999 22952 ts_tablet_manager.cc:1918] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:00.275094 22952 log.cc:1196] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/ce768aef99814340bfcccdfd6608c870
I20260923 22:21:00.275357 22952 ts_tablet_manager.cc:1939] T ce768aef99814340bfcccdfd6608c870 P da1fe9ea0a62445291b0fca08d10578f: Deleting consensus metadata
I20260923 22:21:00.275534 22950 ts_tablet_manager.cc:1905] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:00.275772 22450 catalog_manager.cc:4841] TS da1fe9ea0a62445291b0fca08d10578f (127.4.207.2:44133): tablet ce768aef99814340bfcccdfd6608c870 (table test-table [id=fd3b4e111fcf4a49be2273e0c7ee924f]) successfully deleted
I20260923 22:21:00.277100 22950 ts_tablet_manager.cc:1918] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:00.277155 22950 log.cc:1196] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/ce768aef99814340bfcccdfd6608c870
I20260923 22:21:00.277405 22950 ts_tablet_manager.cc:1939] T ce768aef99814340bfcccdfd6608c870 P 5405a7d5fb5949878ce88530c244c15c: Deleting consensus metadata
W20260923 22:21:00.277565 22450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:40049 (error 108)
I20260923 22:21:00.277845 22449 catalog_manager.cc:4841] TS 5405a7d5fb5949878ce88530c244c15c (127.4.207.3:42059): tablet ce768aef99814340bfcccdfd6608c870 (table test-table [id=fd3b4e111fcf4a49be2273e0c7ee924f]) successfully deleted
I20260923 22:21:00.277869 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 22654
I20260923 22:21:00.278429 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:00.278032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:40049 (local address 127.0.0.1:36270)
0923 22:21:00.278354 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:00.278396 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:40049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301}
W20260923 22:21:00.278878 22450 catalog_manager.cc:4584] TS 6435f9c0cda84df3a8db9232a5565787 (127.4.207.1:40049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce768aef99814340bfcccdfd6608c870: Network error: Client connection negotiation failed: client connection to 127.4.207.1:40049: connect: Connection refused (error 111)
I20260923 22:21:00.283092 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 22785
I20260923 22:21:00.288069 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 22432
2026-09-23T22:21:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (650 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:00 dist-test-slave-7511 krb5kdc[22956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:00 dist-test-slave-7511 krb5kdc[22956](info): set up 2 sockets
Sep 23 22:21:00 dist-test-slave-7511 krb5kdc[22956](info): commencing operation
krb5kdc: starting...
W20260923 22:21:01.371316 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:01 dist-test-slave-7511 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:01.492190 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33365
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:01.571610 22969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.571813 22969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.571834 22969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.572866 22969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:01.572901 22969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.572913 22969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.572924 22969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:01.572935 22969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:01.574213 22969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:01.574385 22969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.574546 22969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:01.579144 22974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.579146 22977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.579135 22975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.579654 22969 server_base.cc:1034] running on GCE node
I20260923 22:21:01.579898 22969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.580228 22969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.581370 22969 hybrid_clock.cc:648] HybridClock initialized: now 1790202061581354 us; error 35 us; skew 500 ppm
Sep 23 22:21:01 dist-test-slave-7511 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:01.589455 22969 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:01.590008 22969 webserver.cc:460] Webserver started at http://127.4.207.62:42965/ using document root <none> and password file <none>
I20260923 22:21:01.590258 22969 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.590335 22969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.590497 22969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.591777 22969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "dd9a0a557b5a44bbb4076e7a50e935dc"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "4d0a92aea7e3e45f34b480dc4191c47e"
server_key_iv: "a69e2c0cbc2f20ceffc6a04e60f585fe"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.592183 22969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "dd9a0a557b5a44bbb4076e7a50e935dc"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "4d0a92aea7e3e45f34b480dc4191c47e"
server_key_iv: "a69e2c0cbc2f20ceffc6a04e60f585fe"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.593621 22969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:01.594421 22984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.594605 22969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:01.594713 22969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "dd9a0a557b5a44bbb4076e7a50e935dc"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "4d0a92aea7e3e45f34b480dc4191c47e"
server_key_iv: "a69e2c0cbc2f20ceffc6a04e60f585fe"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.594789 22969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.606590 22969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.608937 22969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.609100 22969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.613135 22969 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:33365
I20260923 22:21:01.613169 23036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:33365 every 8 connection(s)
I20260923 22:21:01.613521 22969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:01.614405 23037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:01.616750 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 22969
I20260923 22:21:01.616798 23037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc: Bootstrap starting.
I20260923 22:21:01.616876 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:01.617081 4924 external_mini_cluster.cc:1428] Setting key 6720b8848dc9ce751e9eaaf66bbbee54
I20260923 22:21:01.617493 23037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:01.617834 23037 log.cc:824] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:01.618608 23037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc: No bootstrap required, opened a new log
I20260923 22:21:01.620821 23037 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9a0a557b5a44bbb4076e7a50e935dc" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33365 } }
I20260923 22:21:01.620981 23037 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:01.621011 23037 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9a0a557b5a44bbb4076e7a50e935dc, State: Initialized, Role: FOLLOWER
I20260923 22:21:01.621119 23037 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9a0a557b5a44bbb4076e7a50e935dc" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33365 } }
I20260923 22:21:01.621184 23037 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:01.621212 23037 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:01.621253 23037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:01.621778 23037 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9a0a557b5a44bbb4076e7a50e935dc" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33365 } }
I20260923 22:21:01.621901 23037 leader_election.cc:302] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd9a0a557b5a44bbb4076e7a50e935dc; no voters:
I20260923 22:21:01.622071 23037 leader_election.cc:288] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:01.622118 23042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:01.622267 23042 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [term 1 LEADER]: Becoming Leader. State: Replica: dd9a0a557b5a44bbb4076e7a50e935dc, State: Running, Role: LEADER
I20260923 22:21:01.622331 23037 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:01.622424 23042 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9a0a557b5a44bbb4076e7a50e935dc" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33365 } }
I20260923 22:21:01.623656 23044 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd9a0a557b5a44bbb4076e7a50e935dc. Latest consensus state: current_term: 1 leader_uuid: "dd9a0a557b5a44bbb4076e7a50e935dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9a0a557b5a44bbb4076e7a50e935dc" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33365 } } }
I20260923 22:21:01.623732 23044 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:01.623740 23043 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd9a0a557b5a44bbb4076e7a50e935dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd9a0a557b5a44bbb4076e7a50e935dc" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 33365 } } }
Sep 23 22:21:01 dist-test-slave-7511 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:01.624312 23043 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:01.624377 23057 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:01.624418 23057 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:01.624490 23058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:01.624666 23058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:01.626292 23058 catalog_manager.cc:1347] Generated new cluster ID: 93652ca5e5604023bd640ea3d78d1a6b
I20260923 22:21:01.626339 23058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:01.628451 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.617597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.4.207.62:33365)
0923 22:21:01.617785 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.617792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.617817 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:21:01.618447 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.618455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.618508 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.618624 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.619432 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.620335 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.621346 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.621728 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.621804 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.624785 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.624789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.624790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:01.624803 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.626753 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.627125 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.627128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.627128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.627235 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.627431 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.627432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.627433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.627815 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.628010 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.628096 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.628239 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":96,"threads_started":1}
I20260923 22:21:01.652477 23058 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:01.653257 23058 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:01.659409 23058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc: Generated new TSK 0
I20260923 22:21:01.659629 23058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:21:01.689985 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33365
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:01.772758 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.772967 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.772989 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.774109 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.774150 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.774176 23063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:21:01.775702 23063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:01.775960 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.776189 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:01.781224 23069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.781212 23071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.781317 23063 server_base.cc:1034] running on GCE node
W20260923 22:21:01.781224 23068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.781812 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.782099 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.783259 23063 hybrid_clock.cc:648] HybridClock initialized: now 1790202061783237 us; error 35 us; skew 500 ppm
Sep 23 22:21:01 dist-test-slave-7511 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:01.786091 23063 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:21:01.786572 23063 webserver.cc:460] Webserver started at http://127.4.207.1:45997/ using document root <none> and password file <none>
I20260923 22:21:01.786777 23063 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.786818 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.786911 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.788019 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "03fffbc14f887a4fcb7bf128023ee6d4"
server_key_iv: "f0159a615c06f4e0a6eb173b90579c74"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.788335 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "03fffbc14f887a4fcb7bf128023ee6d4"
server_key_iv: "f0159a615c06f4e0a6eb173b90579c74"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.789685 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:01.790432 23078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.790621 23063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:01.790709 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "03fffbc14f887a4fcb7bf128023ee6d4"
server_key_iv: "f0159a615c06f4e0a6eb173b90579c74"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.790763 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.807394 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.809629 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.809734 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.809909 23063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:01.810210 23063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:01.810242 23063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.810294 23063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:01.810318 23063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.815822 23063 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:45083
I20260923 22:21:01.815905 23191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:45083 every 8 connection(s)
I20260923 22:21:01.816215 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:21:01 dist-test-slave-7511 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:01.825155 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 23063
I20260923 22:21:01.825299 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:21:01.825510 4924 external_mini_cluster.cc:1428] Setting key 29d5d1eb65a25065e151db022814ccfe
I20260923 22:21:01.825644 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.816947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:47429 (local address 127.4.207.62:33365)
0923 22:21:01.817000 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.817002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.817207 (+ 205us) server_negotiation.cc:407] Connection header received
0923 22:21:01.817738 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.817740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.817752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.817807 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.818746 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.819249 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.820268 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.820491 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.820580 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.822795 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.822796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.822796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:01.822804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.824355 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.824735 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.824736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.824736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.824787 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.825024 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.825025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.825026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.825136 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.825204 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.825484 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.825559 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:21:01.825834 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.816896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33365 (local address 127.4.207.1:47429)
0923 22:21:01.817080 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.817123 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.817620 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.817876 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.817880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.818434 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:01.818627 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.818632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.819350 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.819351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.820175 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.820177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.820590 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.820591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.820720 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.821264 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:01.821272 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:01.822695 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:01.824429 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:01.824436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:01.824439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:01.824644 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:01.824854 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:01.824855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:01.824855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:01.824958 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:01.825197 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:01.825204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:01.825343 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.825455 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.825606 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260923 22:21:01.826221 23192 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33365
I20260923 22:21:01.826327 23192 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:01.826493 23192 heartbeater.cc:511] Master 127.4.207.62:33365 requested a full tablet report, sending...
I20260923 22:21:01.827090 23000 ts_manager.cc:194] Registered new tserver with Master: 4a338f7dfdaf42dcbd14b4a405b5a68b (127.4.207.1:45083)
I20260923 22:21:01.828047 23000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:47429
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:21:01.850847 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33365
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:01.931018 23197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:01.931365 23197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:01.931495 23197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:01.932677 23197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:01.932744 23197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:01.932787 23197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:21:01.934211 23197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:01.934477 23197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:01.934718 23197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:01.939672 23203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:01.939710 23205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.939738 23197 server_base.cc:1034] running on GCE node
W20260923 22:21:01.939710 23202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:01.940130 23197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:01.940379 23197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:01.941545 23197 hybrid_clock.cc:648] HybridClock initialized: now 1790202061941530 us; error 25 us; skew 500 ppm
Sep 23 22:21:01 dist-test-slave-7511 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:01.944016 23197 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:21:01.944398 23197 webserver.cc:460] Webserver started at http://127.4.207.2:33909/ using document root <none> and password file <none>
I20260923 22:21:01.944566 23197 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:01.944604 23197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:01.944687 23197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:01.945716 23197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "128f5673be9b45d0803a072eedf7e0fa"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "a9d67f5ec119627dd296270618bb16b6"
server_key_iv: "8e4750de8443cd81637ab0becd4ebad2"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.946003 23197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "128f5673be9b45d0803a072eedf7e0fa"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "a9d67f5ec119627dd296270618bb16b6"
server_key_iv: "8e4750de8443cd81637ab0becd4ebad2"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.947196 23197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:01.947911 23212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.948102 23197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.948197 23197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "128f5673be9b45d0803a072eedf7e0fa"
format_stamp: "Formatted at 2026-09-23 22:21:01 on dist-test-slave-7511"
server_key: "a9d67f5ec119627dd296270618bb16b6"
server_key_iv: "8e4750de8443cd81637ab0becd4ebad2"
server_key_version: "encryptionkey@0"
I20260923 22:21:01.948282 23197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:01.973699 23197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:01.976099 23197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:01.976285 23197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:01.976545 23197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:01.976866 23197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:01.976920 23197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.976960 23197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:01.977028 23197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:01.982714 23197 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:35947
I20260923 22:21:01.982761 23325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:35947 every 8 connection(s)
I20260923 22:21:01.983069 23197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:21:01.985704 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 23197
I20260923 22:21:01.985811 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:21:01.986009 4924 external_mini_cluster.cc:1428] Setting key 83fc5574eb334857f8bc0d2c32913c9c
Sep 23 22:21:01 dist-test-slave-7511 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202061, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:01.994192 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.983763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:56489 (local address 127.4.207.62:33365)
0923 22:21:01.983809 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:01.983811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:01.984001 (+ 190us) server_negotiation.cc:407] Connection header received
0923 22:21:01.984497 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:01.984498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:01.984509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:01.984594 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:01.985632 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.986349 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.987556 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.987933 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.988017 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.990841 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:01.990842 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:01.990843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:01.990870 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:01.992553 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.993100 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.993101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.993101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.993161 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:01.993489 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:01.993490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:01.993490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:01.993606 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:01.993666 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:01.994034 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 22:21:01.994116 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:01.994423 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:01.983711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33365 (local address 127.4.207.2:56489)
0923 22:21:01.983862 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:01.983902 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:01.984398 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:01.984671 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:01.984674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:01.985263 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:01.985538 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.985544 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.986424 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.986426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.987482 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:01.987486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:01.988057 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:01.988058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:01.988163 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:01.988777 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:01.988789 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:01.990750 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:01.992648 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:01.992655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:01.992659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:01.993014 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:01.993234 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:01.993235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:01.993236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:01.993380 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:01.993681 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:01.993689 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:01.993869 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 22:21:01.994008 (+ 139us) client_negotiation.cc:240] Negotiation successful
0923 22:21:01.994180 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260923 22:21:01.994974 23326 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33365
I20260923 22:21:01.995096 23326 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:01.995318 23326 heartbeater.cc:511] Master 127.4.207.62:33365 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
I20260923 22:21:01.995800 23000 ts_manager.cc:194] Registered new tserver with Master: 128f5673be9b45d0803a072eedf7e0fa (127.4.207.2:35947)
I20260923 22:21:01.996265 23000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:56489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:21:02.013864 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33365
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:40449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:02.097539 23331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:02.097731 23331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:02.097754 23331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:02.099244 23331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:02.099300 23331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:02.099329 23331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:21:02.100898 23331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:40449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:02.101096 23331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:02.101280 23331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:02.106319 23339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.106410 23331 server_base.cc:1034] running on GCE node
W20260923 22:21:02.106338 23336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:02.106343 23337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.106719 23331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:02.107044 23331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:02.108203 23331 hybrid_clock.cc:648] HybridClock initialized: now 1790202062108189 us; error 25 us; skew 500 ppm
Sep 23 22:21:02 dist-test-slave-7511 krb5kdc[22956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202062, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:02.110560 23331 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:21:02.110996 23331 webserver.cc:460] Webserver started at http://127.4.207.3:46187/ using document root <none> and password file <none>
I20260923 22:21:02.111171 23331 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:02.111207 23331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:02.111285 23331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:02.112277 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "c36ca057c804452dbb37a75148fb6a30"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "b6ff3aae45548128c8e6177963bff2b0"
server_key_iv: "f9814fbe663cc4f35a61c7e0daddd5ea"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.112567 23331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "c36ca057c804452dbb37a75148fb6a30"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "b6ff3aae45548128c8e6177963bff2b0"
server_key_iv: "f9814fbe663cc4f35a61c7e0daddd5ea"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.113848 23331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:21:02.114588 23346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.114769 23331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:02.114851 23331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "c36ca057c804452dbb37a75148fb6a30"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "b6ff3aae45548128c8e6177963bff2b0"
server_key_iv: "f9814fbe663cc4f35a61c7e0daddd5ea"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.114903 23331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:02.125196 23331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:02.127314 23331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:02.127427 23331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:02.127679 23331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:02.128046 23331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:02.128084 23331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.128136 23331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:02.128166 23331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.133939 23331 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:39087
I20260923 22:21:02.134029 23459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:39087 every 8 connection(s)
I20260923 22:21:02.134313 23331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:21:02.138478 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 23331
I20260923 22:21:02.138574 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:21:02.138774 4924 external_mini_cluster.cc:1428] Setting key 9cd510846f7eab02e2cc3d534995d89a
Sep 23 22:21:02 dist-test-slave-7511 krb5kdc[22956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202062, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:02.143824 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.135010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:39769 (local address 127.4.207.62:33365)
0923 22:21:02.135077 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.135079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.135246 (+ 167us) server_negotiation.cc:407] Connection header received
0923 22:21:02.135821 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.135823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.135833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.135893 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:02.136902 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.137406 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.138511 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.138788 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.138871 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.141107 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:02.141108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:02.141109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:02.141115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:02.142522 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:02.142891 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:02.142892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:02.142892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:02.142943 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:02.143218 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:02.143219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:02.143220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:02.143319 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:02.143367 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.143692 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.143761 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 22:21:02.143965 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.134971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:33365 (local address 127.4.207.3:39769)
0923 22:21:02.135131 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.135165 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.135708 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.135969 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.135985 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.136637 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:02.136819 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.136823 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.137507 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.137507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.138423 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.138426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.138948 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.138949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.139106 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.139701 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:02.139709 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:02.141024 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:02.142607 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:02.142612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:02.142614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:02.142822 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:02.143050 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:02.143051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:02.143051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:02.143155 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:02.143373 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:02.143377 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:02.143575 (+ 198us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.143661 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.143779 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260923 22:21:02.144539 23460 heartbeater.cc:347] Connected to a master server at 127.4.207.62:33365
I20260923 22:21:02.144644 23460 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:02.144804 23460 heartbeater.cc:511] Master 127.4.207.62:33365 requested a full tablet report, sending...
I20260923 22:21:02.145195 23000 ts_manager.cc:194] Registered new tserver with Master: c36ca057c804452dbb37a75148fb6a30 (127.4.207.3:39087)
I20260923 22:21:02.145573 23000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:39769
I20260923 22:21:02.150303 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:21:02.157820 23040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.151629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.4.207.62:33365)
0923 22:21:02.151686 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.151688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.151916 (+ 228us) server_negotiation.cc:407] Connection header received
0923 22:21:02.151998 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.151999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.152008 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.152048 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:02.152608 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.153084 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.153785 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.153988 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.154037 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.155241 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:02.155243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:02.155243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:02.155249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:02.156738 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:02.157103 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:02.157104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:02.157104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:02.157156 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:02.157339 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:02.157339 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:02.157340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:02.157465 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:02.157548 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.157644 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.157691 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:21:02.159571 23000 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:21:02.161218 23000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:21:02.167654 23473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.163650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:39087 (local address 127.0.0.1:57974)
0923 22:21:02.164046 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.164054 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.164102 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.164296 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.164297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.164299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.164346 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.164347 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.165815 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.165816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.166710 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.166712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.167352 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.167353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.167427 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.167463 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.167535 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.167584 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":46,"threads_started":1}
I20260923 22:21:02.167665 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.163883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.4.207.3:39087)
0923 22:21:02.164019 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.164024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.164095 (+ 71us) server_negotiation.cc:407] Connection header received
0923 22:21:02.164177 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.164179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.164194 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.164301 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.164729 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.165712 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.166795 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.167256 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.167348 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.167406 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.167549 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.167584 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260923 22:21:02.167977 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.163643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:45083 (local address 127.0.0.1:48388)
0923 22:21:02.163806 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.163846 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.163914 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.164175 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.164178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.164180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.164269 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.164271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.165585 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.165586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.166761 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.166763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.167731 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.167732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.167790 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.167794 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.167838 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.167892 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":39,"thread_start_us":88,"threads_started":1}
I20260923 22:21:02.168160 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.163643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:35947 (local address 127.0.0.1:47300)
0923 22:21:02.164189 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.164197 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.164254 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.164481 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.164482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.164483 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.164530 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.164531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.165437 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.165438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.166760 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.166763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.167730 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.167731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.167993 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.167998 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.168051 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.168095 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":64,"thread_start_us":74,"threads_started":1}
I20260923 22:21:02.168300 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.163830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.4.207.1:45083)
0923 22:21:02.164013 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.164018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.164029 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:21:02.164073 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.164075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.164091 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.164168 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.164368 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.165471 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.166860 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.167575 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.167642 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.167692 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.168150 (+ 458us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.168198 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260923 22:21:02.168467 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.163823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.4.207.2:35947)
0923 22:21:02.164069 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.164073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.164317 (+ 244us) server_negotiation.cc:407] Connection header received
0923 22:21:02.164377 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.164380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.164397 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.164485 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.164591 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.165318 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.166860 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.167575 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.167642 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.167693 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.168317 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.168371 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":71,"threads_started":1}
I20260923 22:21:02.168504 23394 tablet_service.cc:1467] Processing CreateTablet for tablet 989aa22356b74bf381c242950265f749 (DEFAULT_TABLE table=test-table [id=a466fb17e2fd45e69ba1e7f76ace81ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:02.168897 23394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 989aa22356b74bf381c242950265f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:02.168944 23126 tablet_service.cc:1467] Processing CreateTablet for tablet 989aa22356b74bf381c242950265f749 (DEFAULT_TABLE table=test-table [id=a466fb17e2fd45e69ba1e7f76ace81ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:02.169008 23260 tablet_service.cc:1467] Processing CreateTablet for tablet 989aa22356b74bf381c242950265f749 (DEFAULT_TABLE table=test-table [id=a466fb17e2fd45e69ba1e7f76ace81ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:02.169278 23260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 989aa22356b74bf381c242950265f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:02.169278 23126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 989aa22356b74bf381c242950265f749. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:02.171731 23477 tablet_bootstrap.cc:492] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: Bootstrap starting.
I20260923 22:21:02.172360 23477 tablet_bootstrap.cc:654] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:02.172431 23478 tablet_bootstrap.cc:492] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b: Bootstrap starting.
I20260923 22:21:02.173064 23477 log.cc:824] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:02.173110 23478 tablet_bootstrap.cc:654] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:02.173656 23478 log.cc:824] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:02.173866 23477 tablet_bootstrap.cc:492] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: No bootstrap required, opened a new log
I20260923 22:21:02.173936 23477 ts_tablet_manager.cc:1397] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 22:21:02.174000 23479 tablet_bootstrap.cc:492] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Bootstrap starting.
I20260923 22:21:02.174446 23478 tablet_bootstrap.cc:492] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b: No bootstrap required, opened a new log
I20260923 22:21:02.174549 23478 ts_tablet_manager.cc:1397] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:21:02.174595 23479 tablet_bootstrap.cc:654] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:02.175318 23479 log.cc:824] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:02.176101 23479 tablet_bootstrap.cc:492] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: No bootstrap required, opened a new log
I20260923 22:21:02.176019 23477 raft_consensus.cc:348] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.176162 23479 ts_tablet_manager.cc:1397] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:02.176189 23477 raft_consensus.cc:374] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:02.176234 23477 raft_consensus.cc:729] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c36ca057c804452dbb37a75148fb6a30, State: Initialized, Role: FOLLOWER
I20260923 22:21:02.176345 23477 consensus_queue.cc:260] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.176654 23460 heartbeater.cc:503] Master 127.4.207.62:33365 was elected leader, sending a full tablet report...
I20260923 22:21:02.176599 23478 raft_consensus.cc:348] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.176767 23478 raft_consensus.cc:374] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:02.176811 23478 raft_consensus.cc:729] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a338f7dfdaf42dcbd14b4a405b5a68b, State: Initialized, Role: FOLLOWER
I20260923 22:21:02.176643 23477 ts_tablet_manager.cc:1428] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:02.176924 23478 consensus_queue.cc:260] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.177187 23478 ts_tablet_manager.cc:1428] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:02.177201 23192 heartbeater.cc:503] Master 127.4.207.62:33365 was elected leader, sending a full tablet report...
I20260923 22:21:02.178309 23479 raft_consensus.cc:348] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.178454 23479 raft_consensus.cc:374] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:02.178503 23479 raft_consensus.cc:729] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128f5673be9b45d0803a072eedf7e0fa, State: Initialized, Role: FOLLOWER
I20260923 22:21:02.178606 23479 consensus_queue.cc:260] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.178871 23479 ts_tablet_manager.cc:1428] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:02.178884 23326 heartbeater.cc:503] Master 127.4.207.62:33365 was elected leader, sending a full tablet report...
I20260923 22:21:02.221834 23483 raft_consensus.cc:482] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:21:02.221962 23483 raft_consensus.cc:504] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.222265 23483 leader_election.cc:288] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128f5673be9b45d0803a072eedf7e0fa (127.4.207.2:35947), 4a338f7dfdaf42dcbd14b4a405b5a68b (127.4.207.1:45083)
I20260923 22:21:02.225493 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.222561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:42407 (local address 127.4.207.1:45083)
0923 22:21:02.222605 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.222607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.222617 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:02.222705 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.222706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.222718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.222768 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.222981 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.223777 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.224742 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.225128 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.225231 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.225244 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.225387 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.225427 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 22:21:02.225493 23462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.222452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:45083 (local address 127.4.207.3:42407)
0923 22:21:02.222530 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.222544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.222597 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.222769 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.222770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.222771 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.222849 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.222867 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.223856 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.223857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.224649 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.224651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.225232 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.225233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.225312 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.225315 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.225381 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.225421 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 22:21:02.225579 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.222558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:34769 (local address 127.4.207.2:35947)
0923 22:21:02.222607 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.222609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.222828 (+ 219us) server_negotiation.cc:407] Connection header received
0923 22:21:02.222953 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.222955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.222996 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.223042 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.223133 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.223839 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.224809 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.225208 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.225301 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.225310 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.225465 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.225506 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 22:21:02.225617 23486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.222452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:35947 (local address 127.4.207.3:34769)
0923 22:21:02.222749 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.222759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.222841 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.223050 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.223051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.223052 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.223092 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.223093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.223912 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.223913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.224715 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.224717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.225302 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.225303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.225378 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.225381 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.225457 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.225502 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":99,"thread_start_us":145,"threads_started":1}
I20260923 22:21:02.225924 23146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989aa22356b74bf381c242950265f749" candidate_uuid: "c36ca057c804452dbb37a75148fb6a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" is_pre_election: true
I20260923 22:21:02.226049 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989aa22356b74bf381c242950265f749" candidate_uuid: "c36ca057c804452dbb37a75148fb6a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128f5673be9b45d0803a072eedf7e0fa" is_pre_election: true
I20260923 22:21:02.226100 23146 raft_consensus.cc:2454] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c36ca057c804452dbb37a75148fb6a30 in term 0.
I20260923 22:21:02.226177 23280 raft_consensus.cc:2454] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c36ca057c804452dbb37a75148fb6a30 in term 0.
I20260923 22:21:02.226321 23349 leader_election.cc:302] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a338f7dfdaf42dcbd14b4a405b5a68b, c36ca057c804452dbb37a75148fb6a30; no voters:
I20260923 22:21:02.226452 23483 raft_consensus.cc:2781] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:21:02.226503 23483 raft_consensus.cc:482] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:21:02.226536 23483 raft_consensus.cc:3037] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:02.227182 23483 raft_consensus.cc:504] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.227319 23483 leader_election.cc:288] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [CANDIDATE]: Term 1 election: Requested vote from peers 128f5673be9b45d0803a072eedf7e0fa (127.4.207.2:35947), 4a338f7dfdaf42dcbd14b4a405b5a68b (127.4.207.1:45083)
I20260923 22:21:02.227473 23280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989aa22356b74bf381c242950265f749" candidate_uuid: "c36ca057c804452dbb37a75148fb6a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128f5673be9b45d0803a072eedf7e0fa"
I20260923 22:21:02.227483 23146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "989aa22356b74bf381c242950265f749" candidate_uuid: "c36ca057c804452dbb37a75148fb6a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b"
I20260923 22:21:02.227551 23146 raft_consensus.cc:3037] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:02.227550 23280 raft_consensus.cc:3037] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:02.228471 23280 raft_consensus.cc:2454] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c36ca057c804452dbb37a75148fb6a30 in term 1.
I20260923 22:21:02.228471 23146 raft_consensus.cc:2454] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c36ca057c804452dbb37a75148fb6a30 in term 1.
I20260923 22:21:02.228643 23349 leader_election.cc:302] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a338f7dfdaf42dcbd14b4a405b5a68b, c36ca057c804452dbb37a75148fb6a30; no voters:
I20260923 22:21:02.228754 23483 raft_consensus.cc:2781] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:02.228914 23483 raft_consensus.cc:686] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 1 LEADER]: Becoming Leader. State: Replica: c36ca057c804452dbb37a75148fb6a30, State: Running, Role: LEADER
I20260923 22:21:02.229043 23483 consensus_queue.cc:237] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } }
I20260923 22:21:02.229941 23001 catalog_manager.cc:5495] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 reported cstate change: term changed from 0 to 1, leader changed from <none> to c36ca057c804452dbb37a75148fb6a30 (127.4.207.3). New cstate: current_term: 1 leader_uuid: "c36ca057c804452dbb37a75148fb6a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c36ca057c804452dbb37a75148fb6a30" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 39087 } health_report { overall_health: HEALTHY } } }
W20260923 22:21:02.233588 23327 tablet.cc:2369] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:21:02.241024 23476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.238675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.4.207.3:39087)
0923 22:21:02.238729 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.238731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.238760 (+ 29us) server_negotiation.cc:407] Connection header received
0923 22:21:02.238795 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.238796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.238811 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.238863 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:02.238978 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.239423 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.240239 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.240443 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.240528 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.240706 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:02.240798 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:02.240906 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.240936 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.240945 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:21:02.242694 23146 raft_consensus.cc:1261] T 989aa22356b74bf381c242950265f749 P 4a338f7dfdaf42dcbd14b4a405b5a68b [term 1 FOLLOWER]: Refusing update from remote peer c36ca057c804452dbb37a75148fb6a30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:02.242725 23280 raft_consensus.cc:1261] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 1 FOLLOWER]: Refusing update from remote peer c36ca057c804452dbb37a75148fb6a30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:02.243094 23487 consensus_queue.cc:1035] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a338f7dfdaf42dcbd14b4a405b5a68b" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 45083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:02.243227 23483 consensus_queue.cc:1035] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "128f5673be9b45d0803a072eedf7e0fa" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 35947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:02.246270 23492 mvcc.cc:205] Tried to move back new op lower bound from 7332667646943633408 to 7332667646890962944. Current Snapshot: MvccSnapshot[applied={T|T < 7332667646943633408 or (T in {7332667646943633408})}]
I20260923 22:21:02.247713 23000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:02.247810 23000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:02.248569 23000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd9a0a557b5a44bbb4076e7a50e935dc: Sending DeleteTablet for 3 replicas of tablet 989aa22356b74bf381c242950265f749
I20260923 22:21:02.248942 23394 tablet_service.cc:1514] Processing DeleteTablet for tablet 989aa22356b74bf381c242950265f749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:02 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:57974
I20260923 22:21:02.248953 23260 tablet_service.cc:1514] Processing DeleteTablet for tablet 989aa22356b74bf381c242950265f749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:02 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:47300
I20260923 22:21:02.249191 23500 tablet_replica.cc:330] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: stopping tablet replica
I20260923 22:21:02.249176 23126 tablet_service.cc:1514] Processing DeleteTablet for tablet 989aa22356b74bf381c242950265f749 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:02 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:48388
I20260923 22:21:02.249313 23500 raft_consensus.cc:2229] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:02.249413 23500 raft_consensus.cc:2258] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:02.249557 23501 tablet_replica.cc:330] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: stopping tablet replica
I20260923 22:21:02.249636 23501 raft_consensus.cc:2229] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:21:02.249738 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 23063
I20260923 22:21:02.249763 23501 raft_consensus.cc:2258] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:02.250697 23501 ts_tablet_manager.cc:1905] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:02.250735 23500 ts_tablet_manager.cc:1905] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:02.252712 23501 ts_tablet_manager.cc:1918] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:02.252786 23501 log.cc:1196] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/989aa22356b74bf381c242950265f749
I20260923 22:21:02.253054 23501 ts_tablet_manager.cc:1939] T 989aa22356b74bf381c242950265f749 P c36ca057c804452dbb37a75148fb6a30: Deleting consensus metadata
I20260923 22:21:02.253437 22985 catalog_manager.cc:4841] TS c36ca057c804452dbb37a75148fb6a30 (127.4.207.3:39087): tablet 989aa22356b74bf381c242950265f749 (table test-table [id=a466fb17e2fd45e69ba1e7f76ace81ea]) successfully deleted
I20260923 22:21:02.253470 23500 ts_tablet_manager.cc:1918] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:02.253671 23500 log.cc:1196] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/989aa22356b74bf381c242950265f749
I20260923 22:21:02.254048 23500 ts_tablet_manager.cc:1939] T 989aa22356b74bf381c242950265f749 P 128f5673be9b45d0803a072eedf7e0fa: Deleting consensus metadata
I20260923 22:21:02.254539 22988 catalog_manager.cc:4841] TS 128f5673be9b45d0803a072eedf7e0fa (127.4.207.2:35947): tablet 989aa22356b74bf381c242950265f749 (table test-table [id=a466fb17e2fd45e69ba1e7f76ace81ea]) successfully deleted
W20260923 22:21:02.257606 22987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:45083 (error 108)
I20260923 22:21:02.257820 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 23197
I20260923 22:21:02.259284 23472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.258687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:45083 (local address 127.0.0.1:48390)
0923 22:21:02.259202 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.259244 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:45083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492}
W20260923 22:21:02.259459 22987 catalog_manager.cc:4584] TS 4a338f7dfdaf42dcbd14b4a405b5a68b (127.4.207.1:45083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 989aa22356b74bf381c242950265f749: Network error: Client connection negotiation failed: client connection to 127.4.207.1:45083: connect: Connection refused (error 111)
I20260923 22:21:02.263427 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 23331
I20260923 22:21:02.267907 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 22969
2026-09-23T22:21:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1980 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T22:21:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:02Z Disabled control of system clock
I20260923 22:21:02.294857 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39499
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:39499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:02.374145 23507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:02.374383 23507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:02.374403 23507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:02.375640 23507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:02.375684 23507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:02.375696 23507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:02.375710 23507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:02.375763 23507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:02.377116 23507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:39499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:02.377329 23507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:02.377599 23507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:02.382483 23513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:02.382486 23515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.382524 23507 server_base.cc:1034] running on GCE node
W20260923 22:21:02.382506 23512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.382869 23507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:02.383095 23507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:02.384238 23507 hybrid_clock.cc:648] HybridClock initialized: now 1790202062384226 us; error 35 us; skew 500 ppm
I20260923 22:21:02.385542 23507 webserver.cc:460] Webserver started at http://127.4.207.62:45641/ using document root <none> and password file <none>
I20260923 22:21:02.385780 23507 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:02.385846 23507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:02.385977 23507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:02.386988 23507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "05f67d8afd7040598d2e612e8df3b92e"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "7ef92e63b3f7fad11e19d5e4ef8a6489"
server_key_iv: "a63988e6c3dc50e185292a32a1b1832b"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.387346 23507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "05f67d8afd7040598d2e612e8df3b92e"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "7ef92e63b3f7fad11e19d5e4ef8a6489"
server_key_iv: "a63988e6c3dc50e185292a32a1b1832b"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.388674 23507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:02.389475 23521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.389640 23507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:02.389739 23507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "05f67d8afd7040598d2e612e8df3b92e"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "7ef92e63b3f7fad11e19d5e4ef8a6489"
server_key_iv: "a63988e6c3dc50e185292a32a1b1832b"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.389822 23507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:02.400372 23507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:02.400674 23507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:02.400828 23507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:02.404731 23507 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:39499
I20260923 22:21:02.404764 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:39499 every 8 connection(s)
I20260923 22:21:02.405125 23507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:02.406250 23574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:02.408805 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e: Bootstrap starting.
I20260923 22:21:02.409013 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 23507
I20260923 22:21:02.409130 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:02.409332 4924 external_mini_cluster.cc:1428] Setting key 54d3044999ddd0fb3433ffcec5a04ea3
I20260923 22:21:02.409585 23574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:02.410007 23574 log.cc:824] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:02.410790 23574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e: No bootstrap required, opened a new log
I20260923 22:21:02.413219 23574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f67d8afd7040598d2e612e8df3b92e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39499 } }
I20260923 22:21:02.413355 23574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:02.413380 23574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f67d8afd7040598d2e612e8df3b92e, State: Initialized, Role: FOLLOWER
I20260923 22:21:02.413542 23574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f67d8afd7040598d2e612e8df3b92e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39499 } }
I20260923 22:21:02.413625 23574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:02.413668 23574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:02.413720 23574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:02.414440 23574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f67d8afd7040598d2e612e8df3b92e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39499 } }
I20260923 22:21:02.414579 23574 leader_election.cc:302] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05f67d8afd7040598d2e612e8df3b92e; no voters:
I20260923 22:21:02.414775 23574 leader_election.cc:288] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:02.414844 23579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:02.414979 23579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [term 1 LEADER]: Becoming Leader. State: Replica: 05f67d8afd7040598d2e612e8df3b92e, State: Running, Role: LEADER
I20260923 22:21:02.415074 23579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f67d8afd7040598d2e612e8df3b92e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39499 } }
I20260923 22:21:02.415091 23574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:02.415407 23579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05f67d8afd7040598d2e612e8df3b92e. Latest consensus state: current_term: 1 leader_uuid: "05f67d8afd7040598d2e612e8df3b92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f67d8afd7040598d2e612e8df3b92e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39499 } } }
I20260923 22:21:02.415501 23579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:02.415827 23581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05f67d8afd7040598d2e612e8df3b92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05f67d8afd7040598d2e612e8df3b92e" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39499 } } }
I20260923 22:21:02.415864 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.409952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.4.207.62:39499)
0923 22:21:02.410177 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.410187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.410224 (+ 37us) server_negotiation.cc:407] Connection header received
0923 22:21:02.410803 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.410813 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.410894 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.411071 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:02.411256 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.412685 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.413478 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.414027 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.414119 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.414998 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:02.415003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:02.415004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:02.415228 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:02.415279 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:02.415298 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:02.415374 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.415501 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.415602 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260923 22:21:02.415958 23581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:02.416643 23583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:02.417085 23583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:02.418514 23583 catalog_manager.cc:1347] Generated new cluster ID: 07c4f5529d754dc5bc4629398e6c48ce
I20260923 22:21:02.418579 23583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:02.434991 23583 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:02.435784 23583 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:02.443168 23583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e: Generated new TSK 0
I20260923 22:21:02.443396 23583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:02.450397 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39499
--builtin_ntp_servers=127.4.207.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:02.534618 23598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:02.534847 23598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:02.534881 23598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:02.536085 23598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:02.536154 23598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:02.536214 23598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:21:02.537571 23598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:02.537827 23598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:02.538121 23598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:02.543294 23604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.543556 23598 server_base.cc:1034] running on GCE node
W20260923 22:21:02.543279 23603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:02.543300 23606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.543876 23598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:02.544184 23598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:02.545389 23598 hybrid_clock.cc:648] HybridClock initialized: now 1790202062545372 us; error 30 us; skew 500 ppm
I20260923 22:21:02.546881 23598 webserver.cc:460] Webserver started at http://127.4.207.1:44939/ using document root <none> and password file <none>
I20260923 22:21:02.547165 23598 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:02.547245 23598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:02.547403 23598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:02.548625 23598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "fb25c757042e4220a07e8fb226db61db"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "9ac3015436d8e769c56ed0d456ca884e"
server_key_iv: "b9c9dbfd37f51edd88664bea4676e257"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.549039 23598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "fb25c757042e4220a07e8fb226db61db"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "9ac3015436d8e769c56ed0d456ca884e"
server_key_iv: "b9c9dbfd37f51edd88664bea4676e257"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.550462 23598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:02.551241 23612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.551489 23598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:02.551605 23598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "fb25c757042e4220a07e8fb226db61db"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "9ac3015436d8e769c56ed0d456ca884e"
server_key_iv: "b9c9dbfd37f51edd88664bea4676e257"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.551694 23598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:02.574896 23598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:02.575276 23598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:02.575546 23598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:02.575834 23598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:02.576261 23598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:02.576325 23598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.576365 23598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:02.576432 23598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.588282 23598 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:44163
I20260923 22:21:02.588359 23725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:44163 every 8 connection(s)
I20260923 22:21:02.588744 23598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:21:02.594343 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.589533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:48469 (local address 127.4.207.62:39499)
0923 22:21:02.590141 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.590143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.590177 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:02.590418 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.590419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.590430 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.590507 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:02.591167 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.591762 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.592869 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.593083 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.593180 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.594123 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:02.594124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:02.594124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:02.594128 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:02.594141 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:02.594144 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:02.594191 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.594264 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.594284 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577}
I20260923 22:21:02.594617 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.589539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39499 (local address 127.4.207.1:48469)
0923 22:21:02.589790 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.589832 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.590321 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.590564 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.590568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.590576 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:02.591087 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.591092 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.591874 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.591875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.592777 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.592781 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.593188 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.593189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.593321 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.593973 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:02.593974 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:02.594006 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:02.594031 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:02.594199 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:02.594202 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:02.594204 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.594255 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.594340 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":114,"threads_started":1}
I20260923 22:21:02.595204 23726 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39499
I20260923 22:21:02.595316 23726 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:02.595587 23726 heartbeater.cc:511] Master 127.4.207.62:39499 requested a full tablet report, sending...
I20260923 22:21:02.596159 23538 ts_manager.cc:194] Registered new tserver with Master: fb25c757042e4220a07e8fb226db61db (127.4.207.1:44163)
I20260923 22:21:02.596496 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 23598
I20260923 22:21:02.596608 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:21:02.596817 4924 external_mini_cluster.cc:1428] Setting key b0e92b7e1cf2cd43ef44fafe7ce0a264
I20260923 22:21:02.596959 23538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.1:48469
I20260923 22:21:02.597771 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39499
--builtin_ntp_servers=127.4.207.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:02.681766 23729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:02.681968 23729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:02.681990 23729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:02.683162 23729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:02.683205 23729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:02.683230 23729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:21:02.684595 23729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:02.684796 23729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:02.685038 23729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:02.690064 23734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:02.690026 23737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.690071 23729 server_base.cc:1034] running on GCE node
W20260923 22:21:02.690035 23735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.690533 23729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:02.690816 23729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:02.692013 23729 hybrid_clock.cc:648] HybridClock initialized: now 1790202062691993 us; error 28 us; skew 500 ppm
I20260923 22:21:02.693492 23729 webserver.cc:460] Webserver started at http://127.4.207.2:44603/ using document root <none> and password file <none>
I20260923 22:21:02.693783 23729 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:02.693835 23729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:02.693943 23729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:02.695039 23729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "fbe0db57746e4f51acd93f416f1e38d2"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "a9567de237574b43f1beec97ba492fbd"
server_key_iv: "be8646298959edbcef5a564c173a74a3"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.695367 23729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "fbe0db57746e4f51acd93f416f1e38d2"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "a9567de237574b43f1beec97ba492fbd"
server_key_iv: "be8646298959edbcef5a564c173a74a3"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.696825 23729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:02.697576 23743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.697745 23729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:02.697842 23729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "fbe0db57746e4f51acd93f416f1e38d2"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "a9567de237574b43f1beec97ba492fbd"
server_key_iv: "be8646298959edbcef5a564c173a74a3"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.697939 23729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:02.706554 23729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:02.706852 23729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:02.707015 23729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:02.707250 23729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:02.707641 23729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:02.707700 23729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.707744 23729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:02.707775 23729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.713943 23729 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:36903
I20260923 22:21:02.713989 23856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:36903 every 8 connection(s)
I20260923 22:21:02.714344 23729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:21:02.719655 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.715126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:46149 (local address 127.4.207.62:39499)
0923 22:21:02.715176 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.715178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.715498 (+ 320us) server_negotiation.cc:407] Connection header received
0923 22:21:02.716148 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.716149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.716162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.716226 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:02.716865 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.717347 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.718284 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.718470 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.718523 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.719362 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:02.719363 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:02.719363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:02.719367 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:02.719380 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:02.719383 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:02.719428 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.719562 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.719582 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 22:21:02.719861 23859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.715107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39499 (local address 127.4.207.2:46149)
0923 22:21:02.715330 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.715379 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.715934 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.716289 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.716293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.716301 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:02.716790 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.716794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.717425 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.717426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.718204 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.718206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.718544 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.718545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.718654 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.719231 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:02.719232 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:02.719260 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:02.719282 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:02.719429 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:02.719431 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:02.719489 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.719547 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.719637 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260923 22:21:02.720423 23857 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39499
I20260923 22:21:02.720498 23857 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:02.720708 23857 heartbeater.cc:511] Master 127.4.207.62:39499 requested a full tablet report, sending...
I20260923 22:21:02.721093 23538 ts_manager.cc:194] Registered new tserver with Master: fbe0db57746e4f51acd93f416f1e38d2 (127.4.207.2:36903)
I20260923 22:21:02.721448 23538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.2:46149
I20260923 22:21:02.722926 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 23729
I20260923 22:21:02.723021 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:21:02.723176 4924 external_mini_cluster.cc:1428] Setting key 837c57c81d7d6169db94c6bd90630597
I20260923 22:21:02.724032 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39499
--builtin_ntp_servers=127.4.207.20:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:02.803834 23860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:02.804023 23860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:02.804044 23860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:02.805106 23860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:02.805136 23860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:02.805159 23860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:21:02.806583 23860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:39499
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:02.806775 23860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:02.806993 23860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:02.811874 23866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:02.811852 23865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:02.811864 23868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:02.812361 23860 server_base.cc:1034] running on GCE node
I20260923 22:21:02.812582 23860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:02.812824 23860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:02.813997 23860 hybrid_clock.cc:648] HybridClock initialized: now 1790202062814002 us; error 50 us; skew 500 ppm
I20260923 22:21:02.815516 23860 webserver.cc:460] Webserver started at http://127.4.207.3:37719/ using document root <none> and password file <none>
I20260923 22:21:02.815779 23860 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:02.815851 23860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:02.816005 23860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:02.817101 23860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "85039b408a824cbc806d21868646d1f7"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "82984b163b5024a9e3fc9cf4a318fcff"
server_key_iv: "e766edd658a828420eae5ac02e8fa602"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.817462 23860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "85039b408a824cbc806d21868646d1f7"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "82984b163b5024a9e3fc9cf4a318fcff"
server_key_iv: "e766edd658a828420eae5ac02e8fa602"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.818748 23860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:02.819545 23874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.819725 23860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:02.819821 23860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "85039b408a824cbc806d21868646d1f7"
format_stamp: "Formatted at 2026-09-23 22:21:02 on dist-test-slave-7511"
server_key: "82984b163b5024a9e3fc9cf4a318fcff"
server_key_iv: "e766edd658a828420eae5ac02e8fa602"
server_key_version: "encryptionkey@0"
I20260923 22:21:02.819904 23860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:02.830626 23860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:02.830914 23860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:02.831061 23860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:02.831274 23860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:02.831648 23860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:02.831702 23860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.831743 23860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:02.831769 23860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:02.837433 23860 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:33457
I20260923 22:21:02.837473 23987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:33457 every 8 connection(s)
I20260923 22:21:02.837821 23860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:21:02.838666 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 23860
I20260923 22:21:02.838775 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:21:02.839038 4924 external_mini_cluster.cc:1428] Setting key a8b2613c117a0e83c9d6b6de8932d6d5
I20260923 22:21:02.843103 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.838587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:60453 (local address 127.4.207.62:39499)
0923 22:21:02.838645 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.838647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.838905 (+ 258us) server_negotiation.cc:407] Connection header received
0923 22:21:02.839609 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.839610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.839627 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.839699 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:02.840473 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.840945 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.841899 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.842078 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.842127 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.842892 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:02.842893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:02.842893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:02.842897 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:02.842910 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:02.842914 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:02.842950 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.843029 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.843048 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:21:02.843349 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.838547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:39499 (local address 127.4.207.3:60453)
0923 22:21:02.838767 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.838813 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.839486 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.839801 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.839805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.839816 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:02.840389 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.840394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.841031 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.841031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.841811 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.841813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.842151 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.842152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.842255 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.842762 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:02.842763 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:02.842791 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:02.842812 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:02.842961 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:02.842963 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:02.842965 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.843016 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.843092 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":85,"threads_started":1}
I20260923 22:21:02.843909 23988 heartbeater.cc:347] Connected to a master server at 127.4.207.62:39499
I20260923 22:21:02.844014 23988 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:02.844193 23988 heartbeater.cc:511] Master 127.4.207.62:39499 requested a full tablet report, sending...
I20260923 22:21:02.844606 23538 ts_manager.cc:194] Registered new tserver with Master: 85039b408a824cbc806d21868646d1f7 (127.4.207.3:33457)
I20260923 22:21:02.845000 23538 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.3:60453
I20260923 22:21:02.850835 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:21:02.855356 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.852507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.4.207.62:39499)
0923 22:21:02.852566 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.852568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.852665 (+ 97us) server_negotiation.cc:407] Connection header received
0923 22:21:02.852700 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.852701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.852709 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.852756 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:02.852905 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.853351 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.854152 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.854328 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.854384 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.855082 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:02.855083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:02.855083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:02.855088 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:02.855102 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:02.855106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:02.855211 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.855295 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.855305 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 22:21:02.856843 23538 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:21:02.858345 23538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:21:02.864837 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.861007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.4.207.1:44163)
0923 22:21:02.861235 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.861240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.861250 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:02.861289 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.861291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.861307 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.861382 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.861568 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.862226 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.863693 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.864449 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.864532 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.864633 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.864721 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.864759 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260923 22:21:02.864826 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.860527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33457 (local address 127.0.0.1:47742)
0923 22:21:02.860636 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.860675 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.860736 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.861258 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.861259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.861261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.861295 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.861296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.862540 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.862541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.863617 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.863619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.864543 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.864545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.864627 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.864631 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.864689 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.864745 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260923 22:21:02.864831 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.860736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:44163 (local address 127.0.0.1:41974)
0923 22:21:02.861040 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.861051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.861101 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.861455 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.861456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.861458 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.861500 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.861501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.862314 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.862315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.863594 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.863597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.864532 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.864533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.864614 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.864628 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.864709 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.864751 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260923 22:21:02.865664 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.860574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:36903 (local address 127.0.0.1:41688)
0923 22:21:02.860728 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:02.860738 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:02.860791 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:02.861169 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:02.861196 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:02.861199 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:02.861262 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.861264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.863702 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.863703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.864379 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:02.864380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.865398 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.865399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:02.865461 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.865464 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:02.865545 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 22:21:02.865584 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":93,"threads_started":1}
I20260923 22:21:02.865739 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.860736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.4.207.2:36903)
0923 22:21:02.860924 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.860929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.860941 (+ 12us) server_negotiation.cc:407] Connection header received
0923 22:21:02.860985 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.860987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.861002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.861082 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.861342 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.862455 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.864470 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.865173 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.865265 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.865360 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.865573 (+ 213us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.865642 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260923 22:21:02.866039 23644 tablet_service.cc:1467] Processing CreateTablet for tablet d8c54a5111aa473db6dc4d0ccba982ac (DEFAULT_TABLE table=test-table [id=2ec3a4a753f044b99bd023c03333eb86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:02.866379 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:02.860878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.4.207.3:33457)
0923 22:21:02.861100 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:02.861104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:02.861112 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:02.861147 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:02.861149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:02.861164 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:02.861363 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:02.861416 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.862457 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.863693 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:02.864449 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:02.865408 (+ 959us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:02.865484 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:02.866206 (+ 722us) server_negotiation.cc:299] Negotiation successful
0923 22:21:02.866298 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
I20260923 22:21:02.866444 23644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c54a5111aa473db6dc4d0ccba982ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:02.866451 23791 tablet_service.cc:1467] Processing CreateTablet for tablet d8c54a5111aa473db6dc4d0ccba982ac (DEFAULT_TABLE table=test-table [id=2ec3a4a753f044b99bd023c03333eb86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:02.866809 23791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c54a5111aa473db6dc4d0ccba982ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:02.866956 23922 tablet_service.cc:1467] Processing CreateTablet for tablet d8c54a5111aa473db6dc4d0ccba982ac (DEFAULT_TABLE table=test-table [id=2ec3a4a753f044b99bd023c03333eb86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:02.867264 23922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c54a5111aa473db6dc4d0ccba982ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:02.869398 24002 tablet_bootstrap.cc:492] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db: Bootstrap starting.
I20260923 22:21:02.870069 24002 tablet_bootstrap.cc:654] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:02.870177 24003 tablet_bootstrap.cc:492] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Bootstrap starting.
I20260923 22:21:02.870529 24004 tablet_bootstrap.cc:492] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: Bootstrap starting.
I20260923 22:21:02.870548 24002 log.cc:824] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:02.870812 24003 tablet_bootstrap.cc:654] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:02.871126 24003 log.cc:824] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:02.871158 24004 tablet_bootstrap.cc:654] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:02.871500 24004 log.cc:824] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:02.871549 24002 tablet_bootstrap.cc:492] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db: No bootstrap required, opened a new log
I20260923 22:21:02.871675 24002 ts_tablet_manager.cc:1397] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:21:02.871886 24003 tablet_bootstrap.cc:492] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: No bootstrap required, opened a new log
I20260923 22:21:02.871951 24003 ts_tablet_manager.cc:1397] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:02.872223 24004 tablet_bootstrap.cc:492] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: No bootstrap required, opened a new log
I20260923 22:21:02.872303 24004 ts_tablet_manager.cc:1397] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:02.874071 24004 raft_consensus.cc:348] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:02.874239 24004 raft_consensus.cc:374] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:02.874282 24004 raft_consensus.cc:729] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85039b408a824cbc806d21868646d1f7, State: Initialized, Role: FOLLOWER
I20260923 22:21:02.874306 24002 raft_consensus.cc:348] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:02.874418 24004 consensus_queue.cc:260] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:02.874483 24002 raft_consensus.cc:374] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:02.874444 24003 raft_consensus.cc:348] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:02.874536 24002 raft_consensus.cc:729] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb25c757042e4220a07e8fb226db61db, State: Initialized, Role: FOLLOWER
I20260923 22:21:02.874577 24003 raft_consensus.cc:374] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:02.874607 24003 raft_consensus.cc:729] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe0db57746e4f51acd93f416f1e38d2, State: Initialized, Role: FOLLOWER
I20260923 22:21:02.874660 24002 consensus_queue.cc:260] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:02.874701 24004 ts_tablet_manager.cc:1428] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:21:02.874770 24003 consensus_queue.cc:260] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:02.874857 23988 heartbeater.cc:503] Master 127.4.207.62:39499 was elected leader, sending a full tablet report...
I20260923 22:21:02.874950 24002 ts_tablet_manager.cc:1428] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:02.875038 24003 ts_tablet_manager.cc:1428] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:02.875177 23726 heartbeater.cc:503] Master 127.4.207.62:39499 was elected leader, sending a full tablet report...
I20260923 22:21:02.875217 23857 heartbeater.cc:503] Master 127.4.207.62:39499 was elected leader, sending a full tablet report...
W20260923 22:21:02.964934 23858 tablet.cc:2369] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:21:03.010253 24009 raft_consensus.cc:482] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:21:03.010368 24009 raft_consensus.cc:504] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:03.010761 24009 leader_election.cc:288] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85039b408a824cbc806d21868646d1f7 (127.4.207.3:33457), fbe0db57746e4f51acd93f416f1e38d2 (127.4.207.2:36903)
I20260923 22:21:03.013715 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:03.010972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:36903 (local address 127.4.207.1:54653)
0923 22:21:03.011034 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:03.011048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:03.011104 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:03.011300 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:03.011301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:03.011303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:03.011380 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:03.011382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.012079 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.012081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:03.012877 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:03.012879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.013439 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.013440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:03.013514 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:03.013518 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:03.013576 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:21:03.013615 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":16}
I20260923 22:21:03.013720 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:03.011109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:54653 (local address 127.4.207.2:36903)
0923 22:21:03.011164 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:03.011166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:03.011176 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:03.011210 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:03.011211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:03.011223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:03.011279 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:03.011482 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.012002 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.012962 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.013331 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.013415 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:03.013424 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:03.013598 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 22:21:03.013645 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 22:21:03.013794 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:03.011117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:35859 (local address 127.4.207.3:33457)
0923 22:21:03.011173 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:03.011175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:03.011215 (+ 40us) server_negotiation.cc:407] Connection header received
0923 22:21:03.011305 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:03.011305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:03.011314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:03.011373 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:03.011579 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.012027 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.013018 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.013351 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.013480 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:03.013489 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:03.013682 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 22:21:03.013719 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 22:21:03.013859 24011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:03.010997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:33457 (local address 127.4.207.1:35859)
0923 22:21:03.011153 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:03.011164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:03.011230 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:03.011395 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:03.011396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:03.011397 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:03.011505 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:03.011507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.012141 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.012142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:03.012904 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:03.012906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.013511 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.013512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:03.013607 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:03.013611 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:03.013661 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:21:03.013696 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260923 22:21:03.014209 23942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c54a5111aa473db6dc4d0ccba982ac" candidate_uuid: "fb25c757042e4220a07e8fb226db61db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85039b408a824cbc806d21868646d1f7" is_pre_election: true
I20260923 22:21:03.014194 23811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c54a5111aa473db6dc4d0ccba982ac" candidate_uuid: "fb25c757042e4220a07e8fb226db61db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe0db57746e4f51acd93f416f1e38d2" is_pre_election: true
I20260923 22:21:03.014361 23811 raft_consensus.cc:2454] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb25c757042e4220a07e8fb226db61db in term 0.
I20260923 22:21:03.014361 23942 raft_consensus.cc:2454] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb25c757042e4220a07e8fb226db61db in term 0.
I20260923 22:21:03.014599 23616 leader_election.cc:302] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85039b408a824cbc806d21868646d1f7, fb25c757042e4220a07e8fb226db61db; no voters:
I20260923 22:21:03.014739 24009 raft_consensus.cc:2781] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:21:03.014792 24009 raft_consensus.cc:482] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:21:03.014837 24009 raft_consensus.cc:3037] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:03.015594 24009 raft_consensus.cc:504] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:03.015784 24009 leader_election.cc:288] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [CANDIDATE]: Term 1 election: Requested vote from peers 85039b408a824cbc806d21868646d1f7 (127.4.207.3:33457), fbe0db57746e4f51acd93f416f1e38d2 (127.4.207.2:36903)
I20260923 22:21:03.015898 23942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c54a5111aa473db6dc4d0ccba982ac" candidate_uuid: "fb25c757042e4220a07e8fb226db61db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85039b408a824cbc806d21868646d1f7"
I20260923 22:21:03.015923 23811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c54a5111aa473db6dc4d0ccba982ac" candidate_uuid: "fb25c757042e4220a07e8fb226db61db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe0db57746e4f51acd93f416f1e38d2"
I20260923 22:21:03.016016 23942 raft_consensus.cc:3037] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:03.016078 23811 raft_consensus.cc:3037] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:03.016779 23942 raft_consensus.cc:2454] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb25c757042e4220a07e8fb226db61db in term 1.
I20260923 22:21:03.016938 23616 leader_election.cc:302] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85039b408a824cbc806d21868646d1f7, fb25c757042e4220a07e8fb226db61db; no voters:
I20260923 22:21:03.016974 23811 raft_consensus.cc:2454] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb25c757042e4220a07e8fb226db61db in term 1.
I20260923 22:21:03.017023 24009 raft_consensus.cc:2781] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:03.017200 24009 raft_consensus.cc:686] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 1 LEADER]: Becoming Leader. State: Replica: fb25c757042e4220a07e8fb226db61db, State: Running, Role: LEADER
I20260923 22:21:03.017304 24009 consensus_queue.cc:237] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } }
I20260923 22:21:03.018219 23537 catalog_manager.cc:5495] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db reported cstate change: term changed from 0 to 1, leader changed from <none> to fb25c757042e4220a07e8fb226db61db (127.4.207.1). New cstate: current_term: 1 leader_uuid: "fb25c757042e4220a07e8fb226db61db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb25c757042e4220a07e8fb226db61db" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 } health_report { overall_health: UNKNOWN } } }
I20260923 22:21:03.048007 24001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:03.045791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.4.207.1:44163)
0923 22:21:03.045861 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:03.045862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:03.045870 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:03.045906 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:03.045906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:03.045954 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:03.046020 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:03.046147 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.046615 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.047309 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:03.047519 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:03.047573 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:03.047706 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:03.047797 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:03.047876 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:03.047922 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 22:21:03.047933 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 22:21:03.049974 23942 raft_consensus.cc:1261] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 1 FOLLOWER]: Refusing update from remote peer fb25c757042e4220a07e8fb226db61db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:03.049973 23811 raft_consensus.cc:1261] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 1 FOLLOWER]: Refusing update from remote peer fb25c757042e4220a07e8fb226db61db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:03.050348 24012 consensus_queue.cc:1035] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe0db57746e4f51acd93f416f1e38d2" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 36903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:03.050501 24009 consensus_queue.cc:1035] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [LEADER]: Connected to new peer: Peer: permanent_uuid: "85039b408a824cbc806d21868646d1f7" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 33457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:03.052666 24015 mvcc.cc:205] Tried to move back new op lower bound from 7332667650249564160 to 7332667650119540736. Current Snapshot: MvccSnapshot[applied={T|T < 7332667650249564160}]
I20260923 22:21:03.053362 24016 mvcc.cc:205] Tried to move back new op lower bound from 7332667650249564160 to 7332667650119540736. Current Snapshot: MvccSnapshot[applied={T|T < 7332667650249564160}]
I20260923 22:21:03.055207 23538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:03.055303 23538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:03.056067 23538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05f67d8afd7040598d2e612e8df3b92e: Sending DeleteTablet for 3 replicas of tablet d8c54a5111aa473db6dc4d0ccba982ac
I20260923 22:21:03.056396 23791 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c54a5111aa473db6dc4d0ccba982ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:03 UTC) from {username='slave'} at 127.0.0.1:41688
I20260923 22:21:03.056434 23922 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c54a5111aa473db6dc4d0ccba982ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:03 UTC) from {username='slave'} at 127.0.0.1:47742
I20260923 22:21:03.056484 23644 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c54a5111aa473db6dc4d0ccba982ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:03 UTC) from {username='slave'} at 127.0.0.1:41974
I20260923 22:21:03.056685 24027 tablet_replica.cc:330] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db: stopping tablet replica
I20260923 22:21:03.056795 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 23598
I20260923 22:21:03.056823 24027 raft_consensus.cc:2229] T d8c54a5111aa473db6dc4d0ccba982ac P fb25c757042e4220a07e8fb226db61db [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:21:03.056869 24026 tablet_replica.cc:330] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: stopping tablet replica
I20260923 22:21:03.056695 24025 tablet_replica.cc:330] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: stopping tablet replica
I20260923 22:21:03.057773 24025 raft_consensus.cc:2229] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:03.057909 24025 raft_consensus.cc:2258] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:03.058051 24026 raft_consensus.cc:2229] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:03.058146 24026 raft_consensus.cc:2258] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:03.058547 24025 ts_tablet_manager.cc:1905] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:03.058586 24026 ts_tablet_manager.cc:1905] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:03.060556 24026 ts_tablet_manager.cc:1918] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:03.060565 24025 ts_tablet_manager.cc:1918] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:03.060636 24025 log.cc:1196] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/d8c54a5111aa473db6dc4d0ccba982ac
I20260923 22:21:03.060636 24026 log.cc:1196] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/d8c54a5111aa473db6dc4d0ccba982ac
I20260923 22:21:03.060904 24026 ts_tablet_manager.cc:1939] T d8c54a5111aa473db6dc4d0ccba982ac P 85039b408a824cbc806d21868646d1f7: Deleting consensus metadata
I20260923 22:21:03.060904 24025 ts_tablet_manager.cc:1939] T d8c54a5111aa473db6dc4d0ccba982ac P fbe0db57746e4f51acd93f416f1e38d2: Deleting consensus metadata
I20260923 22:21:03.061291 23522 catalog_manager.cc:4841] TS fbe0db57746e4f51acd93f416f1e38d2 (127.4.207.2:36903): tablet d8c54a5111aa473db6dc4d0ccba982ac (table test-table [id=2ec3a4a753f044b99bd023c03333eb86]) successfully deleted
I20260923 22:21:03.061331 23525 catalog_manager.cc:4841] TS 85039b408a824cbc806d21868646d1f7 (127.4.207.3:33457): tablet d8c54a5111aa473db6dc4d0ccba982ac (table test-table [id=2ec3a4a753f044b99bd023c03333eb86]) successfully deleted
W20260923 22:21:03.062660 23525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:44163 (error 108)
I20260923 22:21:03.062913 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 23729
I20260923 22:21:03.065415 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:03.062971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:44163 (local address 127.0.0.1:41996)
0923 22:21:03.063431 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:03.065388 (+ 1957us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:44163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
W20260923 22:21:03.065593 23525 catalog_manager.cc:4584] TS fb25c757042e4220a07e8fb226db61db (127.4.207.1:44163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8c54a5111aa473db6dc4d0ccba982ac: Network error: Client connection negotiation failed: client connection to 127.4.207.1:44163: connect: Connection refused (error 111)
I20260923 22:21:03.068621 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 23860
I20260923 22:21:03.073521 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 23507
2026-09-23T22:21:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (805 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:03 dist-test-slave-7511 krb5kdc[24031](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:03 dist-test-slave-7511 krb5kdc[24031](info): set up 2 sockets
Sep 23 22:21:03 dist-test-slave-7511 krb5kdc[24031](info): commencing operation
krb5kdc: starting...
W20260923 22:21:04.119364 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:04.241863 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:34345
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:34345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:04.320626 24044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:04.320858 24044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:04.320895 24044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:04.322039 24044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:04.322113 24044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:04.322136 24044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:04.322157 24044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:04.322183 24044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:04.323654 24044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:34345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:34345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:04.323953 24044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:04.324186 24044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:04.328989 24049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.329026 24050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.328985 24052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:04.329335 24044 server_base.cc:1034] running on GCE node
I20260923 22:21:04.329605 24044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:04.329851 24044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:04.330991 24044 hybrid_clock.cc:648] HybridClock initialized: now 1790202064330990 us; error 39 us; skew 500 ppm
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:04.333667 24044 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:04.334117 24044 webserver.cc:460] Webserver started at http://127.4.207.62:39041/ using document root <none> and password file <none>
I20260923 22:21:04.334371 24044 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:04.334447 24044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:04.334585 24044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:04.335770 24044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "be8eb3c41d8840d3b028f572fd553a79"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "2edbc47fbb88dfed61f740b6c64f27f4"
server_key_iv: "423f779d258f37df89bcf5a6c9ee1e6f"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.336146 24044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "be8eb3c41d8840d3b028f572fd553a79"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "2edbc47fbb88dfed61f740b6c64f27f4"
server_key_iv: "423f779d258f37df89bcf5a6c9ee1e6f"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.337541 24044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:04.338227 24059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.338395 24044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:04.338508 24044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "be8eb3c41d8840d3b028f572fd553a79"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "2edbc47fbb88dfed61f740b6c64f27f4"
server_key_iv: "423f779d258f37df89bcf5a6c9ee1e6f"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.338584 24044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:04.360392 24044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:04.362861 24044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:04.363118 24044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:04.367323 24044 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:34345
I20260923 22:21:04.367364 24111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:34345 every 8 connection(s)
I20260923 22:21:04.367749 24044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:04.368690 24112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:04.370936 24112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79: Bootstrap starting.
I20260923 22:21:04.371506 24112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:04.371747 24112 log.cc:824] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:04.372424 24112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79: No bootstrap required, opened a new log
I20260923 22:21:04.373947 24112 raft_consensus.cc:348] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8eb3c41d8840d3b028f572fd553a79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34345 } }
I20260923 22:21:04.374086 24112 raft_consensus.cc:374] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:04.374110 24112 raft_consensus.cc:729] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8eb3c41d8840d3b028f572fd553a79, State: Initialized, Role: FOLLOWER
I20260923 22:21:04.374253 24112 consensus_queue.cc:260] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8eb3c41d8840d3b028f572fd553a79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34345 } }
I20260923 22:21:04.374336 24112 raft_consensus.cc:388] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:04.374364 24112 raft_consensus.cc:482] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:04.374398 24112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:04.374876 24112 raft_consensus.cc:504] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8eb3c41d8840d3b028f572fd553a79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34345 } }
I20260923 22:21:04.374959 24112 leader_election.cc:302] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be8eb3c41d8840d3b028f572fd553a79; no voters:
I20260923 22:21:04.375212 24112 leader_election.cc:288] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:04.375234 24115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:04.375393 24115 raft_consensus.cc:686] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [term 1 LEADER]: Becoming Leader. State: Replica: be8eb3c41d8840d3b028f572fd553a79, State: Running, Role: LEADER
I20260923 22:21:04.375499 24115 consensus_queue.cc:237] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8eb3c41d8840d3b028f572fd553a79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34345 } }
I20260923 22:21:04.375556 24112 sys_catalog.cc:564] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:04.375828 24115 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be8eb3c41d8840d3b028f572fd553a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8eb3c41d8840d3b028f572fd553a79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34345 } } }
I20260923 22:21:04.375898 24115 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:04.376027 24115 sys_catalog.cc:455] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be8eb3c41d8840d3b028f572fd553a79. Latest consensus state: current_term: 1 leader_uuid: "be8eb3c41d8840d3b028f572fd553a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be8eb3c41d8840d3b028f572fd553a79" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 34345 } } }
I20260923 22:21:04.376099 24115 sys_catalog.cc:458] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:04.376233 24120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:04.376768 24120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:04.376845 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 24044
I20260923 22:21:04.377022 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:04.377251 4924 external_mini_cluster.cc:1428] Setting key 04f1ee5591a2f5c74bdd6a9cec650dde
I20260923 22:21:04.378545 24120 catalog_manager.cc:1347] Generated new cluster ID: f303929ec7014e03a4080b16b2f4cf0b
I20260923 22:21:04.378599 24120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:04.382594 24120 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:04.383245 24120 catalog_manager.cc:1504] Loading token signing keys...
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:04.387753 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.377892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.4.207.62:34345)
0923 22:21:04.378061 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.378071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.378104 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:21:04.378229 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.378235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.378291 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.378426 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:04.379119 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.380155 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.380935 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.381338 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.381462 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.384082 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:04.384086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:04.384086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:04.384098 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:04.386202 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.386582 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.386585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.386586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.386674 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.386885 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.386886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.386886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.387106 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:04.387260 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.387366 (+ 106us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.387537 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
I20260923 22:21:04.392175 24120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79: Generated new TSK 0
I20260923 22:21:04.392336 24120 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.207.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
Entry for principal kudu/127.4.207.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab.
I20260923 22:21:04.426486 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:34345
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:04.516244 24138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:04.516474 24138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:04.516508 24138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:04.517712 24138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:04.517777 24138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:04.517822 24138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:21:04.519273 24138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.1.keytab
--principal=kudu/127.4.207.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:04.519559 24138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:04.519810 24138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:04.525564 24144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.525755 24146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.525586 24143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:04.526216 24138 server_base.cc:1034] running on GCE node
I20260923 22:21:04.526429 24138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:04.526671 24138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:04.527832 24138 hybrid_clock.cc:648] HybridClock initialized: now 1790202064527819 us; error 29 us; skew 500 ppm
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:04.530387 24138 init.cc:377] Logged in from keytab as kudu/127.4.207.1@KRBTEST.COM (short username kudu)
I20260923 22:21:04.530766 24138 webserver.cc:460] Webserver started at http://127.4.207.1:45107/ using document root <none> and password file <none>
I20260923 22:21:04.531014 24138 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:04.531059 24138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:04.531210 24138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:04.532349 24138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "de4918164972443182ddc182614aa8c6"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "4a63fb10b20584fff853ae1dcc8e5569"
server_key_iv: "58006ab21ebce46f3683d52964b2a167"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.532740 24138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "de4918164972443182ddc182614aa8c6"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "4a63fb10b20584fff853ae1dcc8e5569"
server_key_iv: "58006ab21ebce46f3683d52964b2a167"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.534034 24138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:04.534794 24153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.534996 24138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:04.535084 24138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "de4918164972443182ddc182614aa8c6"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "4a63fb10b20584fff853ae1dcc8e5569"
server_key_iv: "58006ab21ebce46f3683d52964b2a167"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.535167 24138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:04.543041 24138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:04.545297 24138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:04.545436 24138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:04.545708 24138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:04.546005 24138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:04.546036 24138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.546057 24138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:04.546100 24138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.551790 24138 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:39185
I20260923 22:21:04.551851 24266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:39185 every 8 connection(s)
I20260923 22:21:04.552143 24138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.1@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:04.560698 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.552849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:37715 (local address 127.4.207.62:34345)
0923 22:21:04.552910 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.552913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.553117 (+ 204us) server_negotiation.cc:407] Connection header received
0923 22:21:04.553579 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.553580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.553590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.553647 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:04.554550 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.555043 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.556099 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.556295 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.556351 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.558383 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:04.558385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:04.558385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:04.558391 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:04.559620 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.559954 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.559954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.559955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.560004 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.560202 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.560203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.560203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.560297 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:04.560347 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.560564 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.560633 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 22:21:04.560869 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.552813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:34345 (local address 127.4.207.1:37715)
0923 22:21:04.552984 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:04.553025 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:04.553487 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:04.553706 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:04.553709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:04.554257 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:04.554440 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.554444 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.555130 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.555131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.556014 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.556017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.556368 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.556369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.556459 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.556981 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:04.556989 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:04.558274 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:04.559686 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:04.559693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:04.559694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:04.559890 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:04.560060 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:04.560061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:04.560061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:04.560150 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:04.560349 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:04.560353 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:04.560456 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 22:21:04.560538 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 22:21:04.560653 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260923 22:21:04.561362 24267 heartbeater.cc:347] Connected to a master server at 127.4.207.62:34345
I20260923 22:21:04.561475 24267 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:04.561655 24267 heartbeater.cc:511] Master 127.4.207.62:34345 requested a full tablet report, sending...
I20260923 22:21:04.562041 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 24138
I20260923 22:21:04.562150 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:21:04.562255 24076 ts_manager.cc:194] Registered new tserver with Master: de4918164972443182ddc182614aa8c6 (127.4.207.1:39185)
I20260923 22:21:04.562350 4924 external_mini_cluster.cc:1428] Setting key 6049d13a982faed5d2798437e6a47f43
I20260923 22:21:04.562983 24076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.1@KRBTEST.COM'} at 127.4.207.1:37715
WARNING: no policy specified for kudu/127.4.207.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
Entry for principal kudu/127.4.207.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab.
I20260923 22:21:04.587179 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:34345
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:04.668493 24272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:04.668709 24272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:04.668733 24272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:04.669797 24272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:04.669833 24272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:04.669858 24272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:21:04.671267 24272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.2.keytab
--principal=kudu/127.4.207.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:04.671492 24272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:04.671694 24272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:04.676609 24278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.676600 24280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.676630 24277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:04.677094 24272 server_base.cc:1034] running on GCE node
I20260923 22:21:04.677323 24272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:04.677569 24272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:04.678717 24272 hybrid_clock.cc:648] HybridClock initialized: now 1790202064678714 us; error 35 us; skew 500 ppm
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:04.681319 24272 init.cc:377] Logged in from keytab as kudu/127.4.207.2@KRBTEST.COM (short username kudu)
I20260923 22:21:04.681790 24272 webserver.cc:460] Webserver started at http://127.4.207.2:35817/ using document root <none> and password file <none>
I20260923 22:21:04.682097 24272 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:04.682261 24272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:04.682413 24272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:04.683494 24272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "1c4f9b0bc4c04d5faf796705573dd297"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "d2f7f623ccb76e93bfd410d37daf6bf4"
server_key_iv: "62e98364f4e848af7901bdd4145b5849"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.683877 24272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "1c4f9b0bc4c04d5faf796705573dd297"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "d2f7f623ccb76e93bfd410d37daf6bf4"
server_key_iv: "62e98364f4e848af7901bdd4145b5849"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.685165 24272 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:04.685842 24287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.686002 24272 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:04.686096 24272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "1c4f9b0bc4c04d5faf796705573dd297"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "d2f7f623ccb76e93bfd410d37daf6bf4"
server_key_iv: "62e98364f4e848af7901bdd4145b5849"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.686174 24272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:04.702426 24272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:04.704278 24272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:04.704429 24272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:04.704665 24272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:04.704960 24272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:04.705013 24272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.705051 24272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:04.705077 24272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.711024 24272 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:37605
I20260923 22:21:04.711076 24400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:37605 every 8 connection(s)
I20260923 22:21:04.711391 24272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:21:04.711903 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 24272
I20260923 22:21:04.712036 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:21:04.712261 4924 external_mini_cluster.cc:1428] Setting key f8dddc09e69d44b995fe3af9578541de
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.2@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.207.3@KRBTEST.COM; defaulting to no policy
I20260923 22:21:04.721002 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.712163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:49239 (local address 127.4.207.62:34345)
0923 22:21:04.712398 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.712400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.712481 (+ 81us) server_negotiation.cc:407] Connection header received
0923 22:21:04.713119 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.713120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.713131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.713203 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:04.714081 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.714651 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.715714 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.715908 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.715991 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.718099 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:04.718101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:04.718101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:04.718110 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:04.719684 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.720073 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.720074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.720075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.720133 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.720359 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.720360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.720361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.720504 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:04.720580 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.720808 (+ 228us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.720884 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260923 22:21:04.721328 24403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.712157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:34345 (local address 127.4.207.2:49239)
0923 22:21:04.712329 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:04.712375 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:04.713010 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:04.713255 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:04.713258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:04.713799 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:04.713988 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.713993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.714747 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.714748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.715628 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.715631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.716001 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.716002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.716156 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.716680 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:04.716688 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:04.718010 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:04.719757 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:04.719762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:04.719764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:04.719990 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:04.720202 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:04.720202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:04.720202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:04.720291 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:04.720584 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:04.720588 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:04.720691 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 22:21:04.720765 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 22:21:04.720874 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260923 22:21:04.721822 24401 heartbeater.cc:347] Connected to a master server at 127.4.207.62:34345
I20260923 22:21:04.721899 24401 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:04.722085 24401 heartbeater.cc:511] Master 127.4.207.62:34345 requested a full tablet report, sending...
I20260923 22:21:04.722464 24076 ts_manager.cc:194] Registered new tserver with Master: 1c4f9b0bc4c04d5faf796705573dd297 (127.4.207.2:37605)
I20260923 22:21:04.722855 24076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.2@KRBTEST.COM'} at 127.4.207.2:49239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
Entry for principal kudu/127.4.207.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab.
I20260923 22:21:04.738402 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:34345
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:04.817024 24406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:04.817243 24406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:04.817281 24406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:04.818434 24406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:04.818506 24406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:04.818550 24406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:21:04.819976 24406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.3.keytab
--principal=kudu/127.4.207.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:34345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:04.820226 24406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:04.820441 24406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:04.825217 24412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.825223 24414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:04.825233 24411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:04.825595 24406 server_base.cc:1034] running on GCE node
I20260923 22:21:04.825752 24406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:04.825995 24406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:04.827145 24406 hybrid_clock.cc:648] HybridClock initialized: now 1790202064827136 us; error 28 us; skew 500 ppm
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:04.829668 24406 init.cc:377] Logged in from keytab as kudu/127.4.207.3@KRBTEST.COM (short username kudu)
I20260923 22:21:04.830092 24406 webserver.cc:460] Webserver started at http://127.4.207.3:42007/ using document root <none> and password file <none>
I20260923 22:21:04.830323 24406 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:04.830368 24406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:04.830536 24406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:04.831686 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "aff75b84215b43bda10d569e5a373f4f"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "1e52906ed95e7015eb9ed7bd3696b9bc"
server_key_iv: "52fe369573fccb1bce55f3fe84a18943"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.832032 24406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "aff75b84215b43bda10d569e5a373f4f"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "1e52906ed95e7015eb9ed7bd3696b9bc"
server_key_iv: "52fe369573fccb1bce55f3fe84a18943"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.833289 24406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:04.834028 24421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.834196 24406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:04.834292 24406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "aff75b84215b43bda10d569e5a373f4f"
format_stamp: "Formatted at 2026-09-23 22:21:04 on dist-test-slave-7511"
server_key: "1e52906ed95e7015eb9ed7bd3696b9bc"
server_key_iv: "52fe369573fccb1bce55f3fe84a18943"
server_key_version: "encryptionkey@0"
I20260923 22:21:04.834365 24406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:04.855588 24406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:04.857481 24406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:04.857642 24406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:04.857870 24406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:04.858193 24406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:04.858245 24406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.858286 24406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:04.858314 24406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:04.864132 24406 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:44549
I20260923 22:21:04.864202 24534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:44549 every 8 connection(s)
I20260923 22:21:04.864492 24406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
Sep 23 22:21:04 dist-test-slave-7511 krb5kdc[24031](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202064, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.3@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:04.873381 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 24406
I20260923 22:21:04.873499 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:21:04.873710 4924 external_mini_cluster.cc:1428] Setting key 3478ba44f3745a3fc1b4fd971cbc9396
I20260923 22:21:04.874346 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.865184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:49383 (local address 127.4.207.62:34345)
0923 22:21:04.865228 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.865230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.865446 (+ 216us) server_negotiation.cc:407] Connection header received
0923 22:21:04.865961 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.865972 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.865986 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.866035 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:04.866961 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.867558 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.868565 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.868788 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.868875 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.871054 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:04.871056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:04.871056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:04.871063 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:04.872340 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.872721 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.872722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.872723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.872781 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.873043 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.873044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.873044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.873166 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:04.873244 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.874168 (+ 924us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.874256 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:04.874531 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.865170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:34345 (local address 127.4.207.3:49383)
0923 22:21:04.865336 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:04.865369 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:04.865850 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:04.866103 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:04.866107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:04.866677 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:04.866859 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.866863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.867649 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.867650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.868449 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.868451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.868885 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.868886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.869011 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.869529 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:04.869537 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:04.870943 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:04.872410 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:04.872416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:04.872417 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:04.872626 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:04.872849 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:04.872850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:04.872850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:04.872947 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:04.873880 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:04.873885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:04.874026 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 22:21:04.874142 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 22:21:04.874298 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260923 22:21:04.875079 24535 heartbeater.cc:347] Connected to a master server at 127.4.207.62:34345
I20260923 22:21:04.875198 24535 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:04.875401 24535 heartbeater.cc:511] Master 127.4.207.62:34345 requested a full tablet report, sending...
I20260923 22:21:04.875788 24076 ts_manager.cc:194] Registered new tserver with Master: aff75b84215b43bda10d569e5a373f4f (127.4.207.3:44549)
I20260923 22:21:04.876235 24076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.207.3@KRBTEST.COM'} at 127.4.207.3:49383
I20260923 22:21:04.885623 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:21:04.893455 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.887103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.4.207.62:34345)
0923 22:21:04.887170 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.887172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.887207 (+ 35us) server_negotiation.cc:407] Connection header received
0923 22:21:04.887261 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.887262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.887270 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.887317 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:04.888027 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.888574 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.889333 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.889570 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.889630 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.890842 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:04.890843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:04.890844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:04.890850 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:04.892235 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.892610 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.892611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.892612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.892728 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:04.892927 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:04.892928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:04.892928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:04.893068 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:04.893147 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.893290 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.893350 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 22:21:04.895157 24076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:21:04.896854 24076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:21:04.903827 24546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.899360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:44549 (local address 127.0.0.1:60518)
0923 22:21:04.900017 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:04.900030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:04.900140 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:04.900425 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:04.900426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:04.900429 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:04.900471 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.900486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.901699 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.901700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.902600 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.902602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.903483 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.903485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.903576 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.903592 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 22:21:04.903681 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 22:21:04.903736 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":168,"threads_started":1}
I20260923 22:21:04.903839 24549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.899522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.4.207.3:44549)
0923 22:21:04.899711 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.899718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.900106 (+ 388us) server_negotiation.cc:407] Connection header received
0923 22:21:04.900237 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.900239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.900257 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.900316 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:04.900575 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.901601 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.902700 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.903373 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.903482 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.903545 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.903704 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.903746 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260923 22:21:04.904137 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.899354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:37605 (local address 127.0.0.1:36888)
0923 22:21:04.899628 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:04.899674 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:04.899749 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:04.900219 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:04.900220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:04.900223 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:04.900286 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.900288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.901354 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.901355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.902919 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.902921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.903920 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.903921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.903995 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.904000 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:21:04.904035 (+ 35us) client_negotiation.cc:240] Negotiation successful
0923 22:21:04.904078 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":71,"thread_start_us":173,"threads_started":1}
I20260923 22:21:04.904289 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.899354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39185 (local address 127.0.0.1:46124)
0923 22:21:04.899778 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:04.899788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:04.899854 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:04.900141 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:04.900144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:04.900147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:04.900217 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.900219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.901511 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.901512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.902905 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:04.902909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.903948 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.903949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:04.904045 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.904050 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 22:21:04.904098 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 22:21:04.904140 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":6,"thread_start_us":47,"threads_started":1}
I20260923 22:21:04.904299 24550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.899596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.4.207.1:39185)
0923 22:21:04.899792 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.899796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.899839 (+ 43us) server_negotiation.cc:407] Connection header received
0923 22:21:04.899940 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.899942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.899959 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.900038 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:04.900309 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.901390 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.903006 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.903788 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.903858 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.903910 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.904183 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.904223 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260923 22:21:04.904572 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:04.899806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.4.207.2:37605)
0923 22:21:04.900024 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:04.900030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:04.900043 (+ 13us) server_negotiation.cc:407] Connection header received
0923 22:21:04.900087 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:04.900090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:04.900106 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:04.900365 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:04.900439 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.901242 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.903018 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:04.903813 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:04.904350 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:04.904427 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:04.904455 (+ 28us) server_negotiation.cc:299] Negotiation successful
0923 22:21:04.904506 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":118,"threads_started":1}
I20260923 22:21:04.905098 24201 tablet_service.cc:1467] Processing CreateTablet for tablet 26f8346c3d8a4043bbf03c57f200584f (DEFAULT_TABLE table=test-table [id=965810bc7af14f2d90c5a4990325c114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:04.905295 24335 tablet_service.cc:1467] Processing CreateTablet for tablet 26f8346c3d8a4043bbf03c57f200584f (DEFAULT_TABLE table=test-table [id=965810bc7af14f2d90c5a4990325c114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:04.905469 24201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f8346c3d8a4043bbf03c57f200584f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:04.905671 24335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f8346c3d8a4043bbf03c57f200584f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:04.905601 24469 tablet_service.cc:1467] Processing CreateTablet for tablet 26f8346c3d8a4043bbf03c57f200584f (DEFAULT_TABLE table=test-table [id=965810bc7af14f2d90c5a4990325c114]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:04.905882 24469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26f8346c3d8a4043bbf03c57f200584f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:04.908522 24552 tablet_bootstrap.cc:492] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Bootstrap starting.
I20260923 22:21:04.909062 24554 tablet_bootstrap.cc:492] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Bootstrap starting.
I20260923 22:21:04.909334 24552 tablet_bootstrap.cc:654] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:04.909670 24554 tablet_bootstrap.cc:654] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:04.909940 24552 log.cc:824] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:04.910231 24554 log.cc:824] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:04.910557 24553 tablet_bootstrap.cc:492] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6: Bootstrap starting.
I20260923 22:21:04.910789 24552 tablet_bootstrap.cc:492] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: No bootstrap required, opened a new log
I20260923 22:21:04.910876 24552 ts_tablet_manager.cc:1397] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:21:04.911042 24553 tablet_bootstrap.cc:654] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:04.911408 24553 log.cc:824] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:04.911921 24554 tablet_bootstrap.cc:492] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: No bootstrap required, opened a new log
I20260923 22:21:04.911998 24554 ts_tablet_manager.cc:1397] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260923 22:21:04.912729 24552 raft_consensus.cc:348] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:04.912907 24552 raft_consensus.cc:374] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:04.912956 24552 raft_consensus.cc:729] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4f9b0bc4c04d5faf796705573dd297, State: Initialized, Role: FOLLOWER
I20260923 22:21:04.913084 24552 consensus_queue.cc:260] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:04.913389 24552 ts_tablet_manager.cc:1428] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 22:21:04.913426 24401 heartbeater.cc:503] Master 127.4.207.62:34345 was elected leader, sending a full tablet report...
I20260923 22:21:04.913425 24554 raft_consensus.cc:348] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:04.913614 24554 raft_consensus.cc:374] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:04.913661 24554 raft_consensus.cc:729] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff75b84215b43bda10d569e5a373f4f, State: Initialized, Role: FOLLOWER
I20260923 22:21:04.913775 24554 consensus_queue.cc:260] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:04.914060 24554 ts_tablet_manager.cc:1428] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:21:04.914127 24535 heartbeater.cc:503] Master 127.4.207.62:34345 was elected leader, sending a full tablet report...
I20260923 22:21:04.914706 24553 tablet_bootstrap.cc:492] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6: No bootstrap required, opened a new log
I20260923 22:21:04.914793 24553 ts_tablet_manager.cc:1397] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 22:21:04.916447 24553 raft_consensus.cc:348] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:04.916581 24553 raft_consensus.cc:374] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:04.916602 24553 raft_consensus.cc:729] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4918164972443182ddc182614aa8c6, State: Initialized, Role: FOLLOWER
I20260923 22:21:04.916715 24553 consensus_queue.cc:260] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:04.916960 24553 ts_tablet_manager.cc:1428] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 22:21:04.917029 24267 heartbeater.cc:503] Master 127.4.207.62:34345 was elected leader, sending a full tablet report...
W20260923 22:21:04.961985 24402 tablet.cc:2369] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:21:05.052906 24268 tablet.cc:2369] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 22:21:05.115103 24536 tablet.cc:2369] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:21:05.277275 24560 raft_consensus.cc:482] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:21:05.277405 24560 raft_consensus.cc:504] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:05.277791 24560 leader_election.cc:288] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff75b84215b43bda10d569e5a373f4f (127.4.207.3:44549), 1c4f9b0bc4c04d5faf796705573dd297 (127.4.207.2:37605)
I20260923 22:21:05.280848 24549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.278081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:36757 (local address 127.4.207.3:44549)
0923 22:21:05.278140 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.278142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.278184 (+ 42us) server_negotiation.cc:407] Connection header received
0923 22:21:05.278266 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.278267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.278285 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.278350 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:05.278545 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.279036 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.280079 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.280438 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.280524 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.280534 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.280733 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.280774 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 22:21:05.280854 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.278105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:53953 (local address 127.4.207.2:37605)
0923 22:21:05.278158 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.278160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.278288 (+ 128us) server_negotiation.cc:407] Connection header received
0923 22:21:05.278366 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.278367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.278386 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.278447 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:05.278630 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.279090 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.280109 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.280451 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.280540 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.280554 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.280749 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.280786 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:21:05.280871 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.277970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:44549 (local address 127.4.207.1:36757)
0923 22:21:05.278107 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.278120 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:05.278193 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:05.278355 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:05.278356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:05.278358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:05.278437 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.278439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.279125 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.279126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.279970 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.279972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.280536 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.280537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.280647 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.280651 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:05.280724 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 22:21:05.280772 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260923 22:21:05.280874 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.278038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:37605 (local address 127.4.207.1:53953)
0923 22:21:05.278201 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.278219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:05.278300 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:05.278460 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:05.278461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:05.278463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:05.278555 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.278557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.279174 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.279175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.280006 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.280008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.280544 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.280545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.280668 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.280671 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:05.280740 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 22:21:05.280783 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":20,"thread_start_us":77,"threads_started":1}
I20260923 22:21:05.281415 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f8346c3d8a4043bbf03c57f200584f" candidate_uuid: "de4918164972443182ddc182614aa8c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff75b84215b43bda10d569e5a373f4f" is_pre_election: true
I20260923 22:21:05.281416 24355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f8346c3d8a4043bbf03c57f200584f" candidate_uuid: "de4918164972443182ddc182614aa8c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c4f9b0bc4c04d5faf796705573dd297" is_pre_election: true
I20260923 22:21:05.281562 24489 raft_consensus.cc:2454] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de4918164972443182ddc182614aa8c6 in term 0.
I20260923 22:21:05.281577 24355 raft_consensus.cc:2454] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de4918164972443182ddc182614aa8c6 in term 0.
I20260923 22:21:05.281792 24156 leader_election.cc:302] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aff75b84215b43bda10d569e5a373f4f, de4918164972443182ddc182614aa8c6; no voters:
I20260923 22:21:05.281925 24560 raft_consensus.cc:2781] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:21:05.281998 24560 raft_consensus.cc:482] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:21:05.282032 24560 raft_consensus.cc:3037] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:05.282800 24560 raft_consensus.cc:504] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:05.282948 24560 leader_election.cc:288] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [CANDIDATE]: Term 1 election: Requested vote from peers aff75b84215b43bda10d569e5a373f4f (127.4.207.3:44549), 1c4f9b0bc4c04d5faf796705573dd297 (127.4.207.2:37605)
I20260923 22:21:05.283161 24355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f8346c3d8a4043bbf03c57f200584f" candidate_uuid: "de4918164972443182ddc182614aa8c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c4f9b0bc4c04d5faf796705573dd297"
I20260923 22:21:05.283247 24355 raft_consensus.cc:3037] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:05.284021 24355 raft_consensus.cc:2454] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de4918164972443182ddc182614aa8c6 in term 1.
I20260923 22:21:05.284142 24489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26f8346c3d8a4043bbf03c57f200584f" candidate_uuid: "de4918164972443182ddc182614aa8c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff75b84215b43bda10d569e5a373f4f"
I20260923 22:21:05.284235 24157 leader_election.cc:302] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c4f9b0bc4c04d5faf796705573dd297, de4918164972443182ddc182614aa8c6; no voters:
I20260923 22:21:05.284235 24489 raft_consensus.cc:3037] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:05.284338 24560 raft_consensus.cc:2781] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:05.284533 24560 raft_consensus.cc:686] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [term 1 LEADER]: Becoming Leader. State: Replica: de4918164972443182ddc182614aa8c6, State: Running, Role: LEADER
I20260923 22:21:05.284641 24560 consensus_queue.cc:237] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } }
I20260923 22:21:05.285125 24489 raft_consensus.cc:2454] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de4918164972443182ddc182614aa8c6 in term 1.
I20260923 22:21:05.285531 24076 catalog_manager.cc:5495] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to de4918164972443182ddc182614aa8c6 (127.4.207.1). New cstate: current_term: 1 leader_uuid: "de4918164972443182ddc182614aa8c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4918164972443182ddc182614aa8c6" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 39185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 } health_report { overall_health: UNKNOWN } } }
I20260923 22:21:05.349807 24550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.347372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.4.207.1:39185)
0923 22:21:05.347473 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.347475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.347485 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:05.347594 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.347595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.347621 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.347695 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:05.347836 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.348287 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.349065 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.349244 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.349297 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.349434 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:05.349555 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:05.349613 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.349678 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.349695 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 22:21:05.351671 24489 raft_consensus.cc:1261] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 1 FOLLOWER]: Refusing update from remote peer de4918164972443182ddc182614aa8c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:05.351673 24355 raft_consensus.cc:1261] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 1 FOLLOWER]: Refusing update from remote peer de4918164972443182ddc182614aa8c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:05.352037 24563 consensus_queue.cc:1035] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff75b84215b43bda10d569e5a373f4f" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:05.352188 24560 consensus_queue.cc:1035] T 26f8346c3d8a4043bbf03c57f200584f P de4918164972443182ddc182614aa8c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c4f9b0bc4c04d5faf796705573dd297" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 37605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:05.354597 24567 mvcc.cc:205] Tried to move back new op lower bound from 7332667659677532160 to 7332667659406491648. Current Snapshot: MvccSnapshot[applied={T|T < 7332667659677532160}]
I20260923 22:21:05.354825 24566 mvcc.cc:205] Tried to move back new op lower bound from 7332667659677532160 to 7332667659406491648. Current Snapshot: MvccSnapshot[applied={T|T < 7332667659677532160}]
I20260923 22:21:05.357232 24076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:05.357335 24076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:05.358033 24076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be8eb3c41d8840d3b028f572fd553a79: Sending DeleteTablet for 3 replicas of tablet 26f8346c3d8a4043bbf03c57f200584f
I20260923 22:21:05.358402 24201 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f8346c3d8a4043bbf03c57f200584f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:05 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:46124
I20260923 22:21:05.358436 24335 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f8346c3d8a4043bbf03c57f200584f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:05 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:36888
I20260923 22:21:05.358672 24578 tablet_replica.cc:330] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: stopping tablet replica
I20260923 22:21:05.358647 24469 tablet_service.cc:1514] Processing DeleteTablet for tablet 26f8346c3d8a4043bbf03c57f200584f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:05 UTC) from {username='kudu', principal='kudu/127.4.207.62@KRBTEST.COM'} at 127.0.0.1:60518
I20260923 22:21:05.358778 24578 raft_consensus.cc:2229] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:05.358904 24578 raft_consensus.cc:2258] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:05.358979 24579 tablet_replica.cc:330] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: stopping tablet replica
I20260923 22:21:05.359066 24579 raft_consensus.cc:2229] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:05.359169 24579 raft_consensus.cc:2258] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:05.359216 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 24138
I20260923 22:21:05.360149 24578 ts_tablet_manager.cc:1905] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:05.361601 24579 ts_tablet_manager.cc:1905] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:05.362735 24578 ts_tablet_manager.cc:1918] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:05.362828 24578 log.cc:1196] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-1/wal/wals/26f8346c3d8a4043bbf03c57f200584f
I20260923 22:21:05.362815 24579 ts_tablet_manager.cc:1918] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:05.362888 24579 log.cc:1196] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/26f8346c3d8a4043bbf03c57f200584f
I20260923 22:21:05.363121 24578 ts_tablet_manager.cc:1939] T 26f8346c3d8a4043bbf03c57f200584f P 1c4f9b0bc4c04d5faf796705573dd297: Deleting consensus metadata
I20260923 22:21:05.363237 24579 ts_tablet_manager.cc:1939] T 26f8346c3d8a4043bbf03c57f200584f P aff75b84215b43bda10d569e5a373f4f: Deleting consensus metadata
I20260923 22:21:05.363576 24063 catalog_manager.cc:4841] TS 1c4f9b0bc4c04d5faf796705573dd297 (127.4.207.2:37605): tablet 26f8346c3d8a4043bbf03c57f200584f (table test-table [id=965810bc7af14f2d90c5a4990325c114]) successfully deleted
I20260923 22:21:05.363670 24062 catalog_manager.cc:4841] TS aff75b84215b43bda10d569e5a373f4f (127.4.207.3:44549): tablet 26f8346c3d8a4043bbf03c57f200584f (table test-table [id=965810bc7af14f2d90c5a4990325c114]) successfully deleted
W20260923 22:21:05.365013 24060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:39185 (error 108)
I20260923 22:21:05.365240 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 24272
I20260923 22:21:05.365846 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.365351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:39185 (local address 127.0.0.1:46132)
0923 22:21:05.365772 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.365807 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403}
W20260923 22:21:05.366312 24060 catalog_manager.cc:4584] TS de4918164972443182ddc182614aa8c6 (127.4.207.1:39185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26f8346c3d8a4043bbf03c57f200584f: Network error: Client connection negotiation failed: client connection to 127.4.207.1:39185: connect: Connection refused (error 111)
I20260923 22:21:05.370124 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 24406
I20260923 22:21:05.374713 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 24044
2026-09-23T22:21:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2302 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T22:21:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:05Z Disabled control of system clock
I20260923 22:21:05.402385 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:43855
--webserver_interface=127.4.207.62
--webserver_port=0
--builtin_ntp_servers=127.4.207.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:43855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:05.483693 24584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:05.483913 24584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:05.483944 24584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:05.485039 24584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:05.485080 24584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:05.485095 24584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:05.485108 24584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:05.485121 24584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:05.486402 24584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:43855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:43855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:05.486611 24584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:05.486858 24584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:05.491520 24592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:05.491541 24589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:05.491539 24590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:05.492151 24584 server_base.cc:1034] running on GCE node
I20260923 22:21:05.492404 24584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:05.492683 24584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:05.493863 24584 hybrid_clock.cc:648] HybridClock initialized: now 1790202065493847 us; error 27 us; skew 500 ppm
I20260923 22:21:05.495157 24584 webserver.cc:460] Webserver started at http://127.4.207.62:42331/ using document root <none> and password file <none>
I20260923 22:21:05.495395 24584 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:05.495502 24584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:05.495658 24584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:05.496701 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "061446dda92f42cb93be5391d26c7b32"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "dce2ce1fc3fdb078a123cbca8a8da849"
server_key_iv: "8c0f925bd2df304ece6308ebd2a59af4"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.497061 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "061446dda92f42cb93be5391d26c7b32"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "dce2ce1fc3fdb078a123cbca8a8da849"
server_key_iv: "8c0f925bd2df304ece6308ebd2a59af4"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.498315 24584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:05.499020 24598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.499197 24584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:05.499295 24584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "061446dda92f42cb93be5391d26c7b32"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "dce2ce1fc3fdb078a123cbca8a8da849"
server_key_iv: "8c0f925bd2df304ece6308ebd2a59af4"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.499377 24584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:05.509744 24584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:05.510039 24584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:05.510186 24584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:05.514046 24584 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:43855
I20260923 22:21:05.514073 24650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:43855 every 8 connection(s)
I20260923 22:21:05.514403 24584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:05.515292 24651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:05.516518 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 24584
I20260923 22:21:05.516642 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:05.516815 4924 external_mini_cluster.cc:1428] Setting key f6c8e435e9d79a528b09e1e0a0a78263
I20260923 22:21:05.517979 24651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32: Bootstrap starting.
I20260923 22:21:05.518673 24651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:05.518986 24651 log.cc:824] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:05.519706 24651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32: No bootstrap required, opened a new log
I20260923 22:21:05.521831 24651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061446dda92f42cb93be5391d26c7b32" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43855 } }
I20260923 22:21:05.522014 24651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:05.522064 24651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 061446dda92f42cb93be5391d26c7b32, State: Initialized, Role: FOLLOWER
I20260923 22:21:05.522205 24651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061446dda92f42cb93be5391d26c7b32" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43855 } }
I20260923 22:21:05.522306 24651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:05.522351 24651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:05.522397 24651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:05.522840 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.517334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.4.207.62:43855)
0923 22:21:05.517546 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.517555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.517589 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:05.518121 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.518130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.518202 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.518360 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:05.518522 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.519901 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.520680 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.521316 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.521455 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.522117 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:05.522121 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:05.522121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:05.522303 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:05.522361 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:05.522377 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:05.522442 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.522538 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.522642 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260923 22:21:05.522997 24651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061446dda92f42cb93be5391d26c7b32" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43855 } }
I20260923 22:21:05.523099 24651 leader_election.cc:302] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 061446dda92f42cb93be5391d26c7b32; no voters:
I20260923 22:21:05.523263 24651 leader_election.cc:288] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:05.523293 24656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:05.523396 24656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [term 1 LEADER]: Becoming Leader. State: Replica: 061446dda92f42cb93be5391d26c7b32, State: Running, Role: LEADER
I20260923 22:21:05.523538 24651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:05.523638 24656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061446dda92f42cb93be5391d26c7b32" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43855 } }
I20260923 22:21:05.523942 24657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "061446dda92f42cb93be5391d26c7b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061446dda92f42cb93be5391d26c7b32" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43855 } } }
I20260923 22:21:05.523942 24658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 061446dda92f42cb93be5391d26c7b32. Latest consensus state: current_term: 1 leader_uuid: "061446dda92f42cb93be5391d26c7b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "061446dda92f42cb93be5391d26c7b32" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43855 } } }
I20260923 22:21:05.524050 24657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:05.524099 24658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:05.524289 24663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:05.524762 24663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:05.526294 24663 catalog_manager.cc:1347] Generated new cluster ID: a43469626e514fb5834838a2aaf4b9de
I20260923 22:21:05.526348 24663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:05.538264 24663 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:05.538808 24663 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:05.543860 24663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32: Generated new TSK 0
I20260923 22:21:05.544085 24663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:05.552194 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.1:0
--local_ip_for_outbound_sockets=127.4.207.1
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43855
--builtin_ntp_servers=127.4.207.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:05.633257 24675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:05.633462 24675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:05.633484 24675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:05.634624 24675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:05.634662 24675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:05.634686 24675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.1
I20260923 22:21:05.636031 24675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.207.1
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.1
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:05.636272 24675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:05.636598 24675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:05.642297 24680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:05.642313 24681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:05.642315 24683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:05.642880 24675 server_base.cc:1034] running on GCE node
I20260923 22:21:05.643119 24675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:05.643373 24675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:05.644522 24675 hybrid_clock.cc:648] HybridClock initialized: now 1790202065644504 us; error 34 us; skew 500 ppm
I20260923 22:21:05.645756 24675 webserver.cc:460] Webserver started at http://127.4.207.1:39923/ using document root <none> and password file <none>
I20260923 22:21:05.646020 24675 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:05.646065 24675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:05.646201 24675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:05.647310 24675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/data/instance:
uuid: "dc39f5bfc9044094a62a9c5632e2f9c7"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "85bed455f4db1ff3d5c48fadc5c58812"
server_key_iv: "98f1c954bae8370c54ebffbd4fd502e3"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.647714 24675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance:
uuid: "dc39f5bfc9044094a62a9c5632e2f9c7"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "85bed455f4db1ff3d5c48fadc5c58812"
server_key_iv: "98f1c954bae8370c54ebffbd4fd502e3"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.649044 24675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:05.649765 24689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.649945 24675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:05.650041 24675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/wal
uuid: "dc39f5bfc9044094a62a9c5632e2f9c7"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "85bed455f4db1ff3d5c48fadc5c58812"
server_key_iv: "98f1c954bae8370c54ebffbd4fd502e3"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.650127 24675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:05.659857 24675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:05.660136 24675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:05.660279 24675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:05.660493 24675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:05.660812 24675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:05.660866 24675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.660904 24675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:05.660930 24675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.666746 24675 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.1:42549
I20260923 22:21:05.666780 24802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.1:42549 every 8 connection(s)
I20260923 22:21:05.667106 24675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/data/info.pb
I20260923 22:21:05.672226 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.667905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.1:55011 (local address 127.4.207.62:43855)
0923 22:21:05.667954 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.667956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.668148 (+ 192us) server_negotiation.cc:407] Connection header received
0923 22:21:05.668658 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.668660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.668689 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.668764 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:05.669440 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.669954 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.670886 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.671085 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.671145 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.671978 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:05.671980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:05.671980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:05.671984 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:05.671999 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:05.672003 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:05.672054 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.672141 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.672164 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:05.672396 24805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.667825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43855 (local address 127.4.207.1:55011)
0923 22:21:05.668012 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.668054 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:05.668537 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:05.668842 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:05.668846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:05.668854 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:05.669343 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.669348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.670039 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.670040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.670810 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.670812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.671174 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.671175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.671278 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.671849 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:05.671849 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:05.671876 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:05.671898 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:05.672064 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:05.672068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:05.672070 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:21:05.672121 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 22:21:05.672208 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260923 22:21:05.672890 24803 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43855
I20260923 22:21:05.672977 24803 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:05.673110 24803 heartbeater.cc:511] Master 127.4.207.62:43855 requested a full tablet report, sending...
I20260923 22:21:05.673620 24615 ts_manager.cc:194] Registered new tserver with Master: dc39f5bfc9044094a62a9c5632e2f9c7 (127.4.207.1:42549)
I20260923 22:21:05.674268 24615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.1:55011
I20260923 22:21:05.676915 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 24675
I20260923 22:21:05.677064 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-0/wal/instance
I20260923 22:21:05.677274 4924 external_mini_cluster.cc:1428] Setting key af94fe7fdef135d9ffeea587efefa238
I20260923 22:21:05.678160 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.2:0
--local_ip_for_outbound_sockets=127.4.207.2
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43855
--builtin_ntp_servers=127.4.207.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:05.756636 24806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:05.756867 24806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:05.756901 24806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:05.758067 24806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:05.758133 24806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:05.758180 24806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.2
I20260923 22:21:05.759527 24806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.207.2
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.2
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:05.759821 24806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:05.760139 24806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:05.765551 24811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:05.765571 24812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:05.765568 24814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:05.765600 24806 server_base.cc:1034] running on GCE node
I20260923 22:21:05.765983 24806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:05.766201 24806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:05.767338 24806 hybrid_clock.cc:648] HybridClock initialized: now 1790202065767330 us; error 34 us; skew 500 ppm
I20260923 22:21:05.768733 24806 webserver.cc:460] Webserver started at http://127.4.207.2:45909/ using document root <none> and password file <none>
I20260923 22:21:05.768939 24806 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:05.768978 24806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:05.769088 24806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:05.770084 24806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/data/instance:
uuid: "a631235c8c16432e961b688f859f0988"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "9ad4cdee7d38256c70e4f0c2635ab904"
server_key_iv: "b9f15daa23f4de74efc0d47115052b1f"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.770381 24806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance:
uuid: "a631235c8c16432e961b688f859f0988"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "9ad4cdee7d38256c70e4f0c2635ab904"
server_key_iv: "b9f15daa23f4de74efc0d47115052b1f"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.771626 24806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:05.772295 24820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.772541 24806 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 22:21:05.772612 24806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/wal
uuid: "a631235c8c16432e961b688f859f0988"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "9ad4cdee7d38256c70e4f0c2635ab904"
server_key_iv: "b9f15daa23f4de74efc0d47115052b1f"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.772691 24806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:05.788399 24806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:05.788733 24806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:05.788888 24806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:05.789127 24806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:05.789458 24806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:05.789512 24806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.789546 24806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:05.789577 24806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.795353 24806 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.2:38701
I20260923 22:21:05.795392 24933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.2:38701 every 8 connection(s)
I20260923 22:21:05.795805 24806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/data/info.pb
I20260923 22:21:05.801221 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.796582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.2:48359 (local address 127.4.207.62:43855)
0923 22:21:05.796641 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.796642 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.796879 (+ 237us) server_negotiation.cc:407] Connection header received
0923 22:21:05.797417 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.797418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.797431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.797499 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:05.798203 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.798718 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.799840 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.800039 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.800117 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.800889 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:05.800891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:05.800891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:05.800896 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:05.800912 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:05.800916 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:05.801018 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.801114 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.801141 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 22:21:05.801468 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.796550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43855 (local address 127.4.207.2:48359)
0923 22:21:05.796760 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.796793 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:05.797309 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:05.797584 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:05.797589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:05.797605 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:05.798103 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.798108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.798819 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.798820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.799740 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.799743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.800122 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.800123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.800255 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.800751 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:05.800751 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:05.800779 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:05.800802 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:05.801021 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:05.801027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:05.801030 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:05.801116 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 22:21:05.801223 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":91,"threads_started":1}
I20260923 22:21:05.801996 24934 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43855
I20260923 22:21:05.802076 24934 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:05.802284 24934 heartbeater.cc:511] Master 127.4.207.62:43855 requested a full tablet report, sending...
I20260923 22:21:05.802676 24615 ts_manager.cc:194] Registered new tserver with Master: a631235c8c16432e961b688f859f0988 (127.4.207.2:38701)
I20260923 22:21:05.802913 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 24806
I20260923 22:21:05.803058 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-1/wal/instance
I20260923 22:21:05.803145 24615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.2:48359
I20260923 22:21:05.803290 4924 external_mini_cluster.cc:1428] Setting key b0fee7c457120f465acedae84970932e
I20260923 22:21:05.804214 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.207.3:0
--local_ip_for_outbound_sockets=127.4.207.3
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43855
--builtin_ntp_servers=127.4.207.20:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 22:21:05.882732 24937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:05.882963 24937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:05.882998 24937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:05.884176 24937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:05.884240 24937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:05.884281 24937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.207.3
I20260923 22:21:05.885602 24937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.207.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.207.3
--webserver_port=0
--tserver_master_addrs=127.4.207.62:43855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.207.3
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:05.885849 24937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:05.886132 24937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:05.890974 24942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:05.891055 24943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:05.891212 24937 server_base.cc:1034] running on GCE node
W20260923 22:21:05.890978 24945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:05.891583 24937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:05.891836 24937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:05.892978 24937 hybrid_clock.cc:648] HybridClock initialized: now 1790202065892947 us; error 49 us; skew 500 ppm
I20260923 22:21:05.894378 24937 webserver.cc:460] Webserver started at http://127.4.207.3:33657/ using document root <none> and password file <none>
I20260923 22:21:05.894618 24937 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:05.894661 24937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:05.894797 24937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:05.895929 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/data/instance:
uuid: "523cdabf532849248a877a8abd5dd328"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "b49c2f1d6fcab5e62e609e0c1adb18bc"
server_key_iv: "dae8541f9fb8243f51dc8c16642484f1"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.896274 24937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance:
uuid: "523cdabf532849248a877a8abd5dd328"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "b49c2f1d6fcab5e62e609e0c1adb18bc"
server_key_iv: "dae8541f9fb8243f51dc8c16642484f1"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.897544 24937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:05.898232 24951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.898418 24937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:05.898492 24937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/wal
uuid: "523cdabf532849248a877a8abd5dd328"
format_stamp: "Formatted at 2026-09-23 22:21:05 on dist-test-slave-7511"
server_key: "b49c2f1d6fcab5e62e609e0c1adb18bc"
server_key_iv: "dae8541f9fb8243f51dc8c16642484f1"
server_key_version: "encryptionkey@0"
I20260923 22:21:05.898583 24937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:05.911684 24937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:05.911964 24937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:05.912108 24937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:05.912324 24937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 22:21:05.912701 24937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 22:21:05.912763 24937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.912803 24937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 22:21:05.912832 24937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:05.918576 24937 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.3:35553
I20260923 22:21:05.918623 25064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.3:35553 every 8 connection(s)
I20260923 22:21:05.918936 24937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/data/info.pb
I20260923 22:21:05.923733 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.919664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:34653 (local address 127.4.207.62:43855)
0923 22:21:05.919704 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.919706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.919924 (+ 218us) server_negotiation.cc:407] Connection header received
0923 22:21:05.920382 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.920383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.920396 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.920468 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:05.921065 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.921534 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.922427 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.922643 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.922707 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.923489 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:05.923491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:05.923491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:05.923496 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:05.923511 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:05.923515 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:05.923564 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.923634 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.923662 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:05.923938 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.919653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:43855 (local address 127.4.207.3:34653)
0923 22:21:05.919796 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.919829 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:05.920265 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:05.920519 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:05.920522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:05.920530 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:05.920988 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.920992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.921607 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.921608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.922328 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.922329 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.922722 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.922722 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.922827 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.923312 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 22:21:05.923312 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:05.923338 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 22:21:05.923358 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:05.923567 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:05.923570 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:05.923572 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 22:21:05.923622 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:21:05.923695 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260923 22:21:05.924461 25065 heartbeater.cc:347] Connected to a master server at 127.4.207.62:43855
I20260923 22:21:05.924535 25065 heartbeater.cc:464] Registering TS with master...
I20260923 22:21:05.924722 25065 heartbeater.cc:511] Master 127.4.207.62:43855 requested a full tablet report, sending...
I20260923 22:21:05.925128 24615 ts_manager.cc:194] Registered new tserver with Master: 523cdabf532849248a877a8abd5dd328 (127.4.207.3:35553)
I20260923 22:21:05.925446 24615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.207.3:34653
I20260923 22:21:05.928637 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 24937
I20260923 22:21:05.928731 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/wal/instance
I20260923 22:21:05.928884 4924 external_mini_cluster.cc:1428] Setting key 9eb6053745e09fcc044ab42630f13296
I20260923 22:21:05.929826 4924 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 22:21:05.934037 24654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.931104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46646 (local address 127.4.207.62:43855)
0923 22:21:05.931140 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.931141 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.931287 (+ 146us) server_negotiation.cc:407] Connection header received
0923 22:21:05.931369 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.931370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.931380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.931442 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:05.931647 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.932126 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.932860 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.933036 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.933103 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.933787 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:05.933788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:05.933788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:05.933793 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:05.933805 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:05.933809 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:05.933874 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.933934 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.933960 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:21:05.935683 24615 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 22:21:05.937049 24615 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 22:21:05.942839 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.938996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.3:35553 (local address 127.0.0.1:33046)
0923 22:21:05.939152 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.939188 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:05.939247 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:05.939601 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:05.939603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:05.939606 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:05.939664 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.939666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.940585 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.940586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.941811 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.941814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.942561 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.942561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.942637 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.942649 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 22:21:05.942721 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 22:21:05.942771 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260923 22:21:05.942853 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.939106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.4.207.3:35553)
0923 22:21:05.939349 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.939352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.939362 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:05.939399 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.939401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.939427 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.939525 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:05.939792 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.940491 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.941914 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.942464 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.942559 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.942646 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.942735 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.942770 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":74,"threads_started":1}
I20260923 22:21:05.943127 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.939187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:38701 (local address 127.0.0.1:36532)
0923 22:21:05.939393 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.939415 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:05.939494 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:05.939800 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:05.939801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:05.939802 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:05.939835 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.939837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.941019 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.941020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.941834 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.941836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.942866 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.942867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.942930 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.942934 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:05.942984 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:21:05.943030 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":67,"threads_started":1}
I20260923 22:21:05.943349 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.939375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.4.207.2:38701)
0923 22:21:05.939608 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.939612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.939620 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:05.939656 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.939657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.939672 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.939737 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:05.939904 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.940960 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.941936 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.942672 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.942744 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.942829 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.943231 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.943270 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":118,"threads_started":1}
I20260923 22:21:05.943349 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.939123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.4.207.1:42549)
0923 22:21:05.939280 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:05.939285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:05.939440 (+ 155us) server_negotiation.cc:407] Connection header received
0923 22:21:05.939519 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:05.939522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:05.939547 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:05.939615 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:05.939764 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.940853 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.942076 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.942760 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.942817 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.942888 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:05.943168 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 22:21:05.943269 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260923 22:21:05.943744 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:05.939011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:42549 (local address 127.0.0.1:38366)
0923 22:21:05.939296 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:05.939308 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:05.939372 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:05.939614 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:05.939615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:05.939617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:05.939680 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.939681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.940947 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.940947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.941999 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:05.942001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:05.942916 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:05.942917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:05.942989 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:05.942992 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 22:21:05.943641 (+ 649us) client_negotiation.cc:240] Negotiation successful
0923 22:21:05.943684 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":15,"thread_start_us":87,"threads_started":1}
I20260923 22:21:05.943715 24999 tablet_service.cc:1467] Processing CreateTablet for tablet 938318ed3db646a89d5b5544bde5c04c (DEFAULT_TABLE table=test-table [id=3a307c17ed6840e9a92c6907b8a20040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:05.943994 24868 tablet_service.cc:1467] Processing CreateTablet for tablet 938318ed3db646a89d5b5544bde5c04c (DEFAULT_TABLE table=test-table [id=3a307c17ed6840e9a92c6907b8a20040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:05.944126 24999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938318ed3db646a89d5b5544bde5c04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:05.944319 24868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938318ed3db646a89d5b5544bde5c04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:05.944370 24737 tablet_service.cc:1467] Processing CreateTablet for tablet 938318ed3db646a89d5b5544bde5c04c (DEFAULT_TABLE table=test-table [id=3a307c17ed6840e9a92c6907b8a20040]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 22:21:05.944692 24737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938318ed3db646a89d5b5544bde5c04c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:05.946619 25079 tablet_bootstrap.cc:492] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988: Bootstrap starting.
I20260923 22:21:05.947294 25080 tablet_bootstrap.cc:492] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: Bootstrap starting.
I20260923 22:21:05.947361 25079 tablet_bootstrap.cc:654] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:05.947713 25079 log.cc:824] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:05.947712 25081 tablet_bootstrap.cc:492] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7: Bootstrap starting.
I20260923 22:21:05.948015 25080 tablet_bootstrap.cc:654] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:05.948312 25080 log.cc:824] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:05.948365 25081 tablet_bootstrap.cc:654] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:05.948657 25079 tablet_bootstrap.cc:492] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988: No bootstrap required, opened a new log
I20260923 22:21:05.948715 25081 log.cc:824] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:05.948755 25079 ts_tablet_manager.cc:1397] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 22:21:05.949052 25080 tablet_bootstrap.cc:492] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: No bootstrap required, opened a new log
I20260923 22:21:05.949103 25080 ts_tablet_manager.cc:1397] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:05.949517 25081 tablet_bootstrap.cc:492] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7: No bootstrap required, opened a new log
I20260923 22:21:05.949584 25081 ts_tablet_manager.cc:1397] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 22:21:05.950906 25079 raft_consensus.cc:348] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:05.951056 25079 raft_consensus.cc:374] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:05.951095 25079 raft_consensus.cc:729] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a631235c8c16432e961b688f859f0988, State: Initialized, Role: FOLLOWER
I20260923 22:21:05.951217 25079 consensus_queue.cc:260] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:05.951491 25079 ts_tablet_manager.cc:1428] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 22:21:05.951599 24934 heartbeater.cc:503] Master 127.4.207.62:43855 was elected leader, sending a full tablet report...
I20260923 22:21:05.951552 25080 raft_consensus.cc:348] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:05.951684 25080 raft_consensus.cc:374] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:05.951712 25080 raft_consensus.cc:729] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523cdabf532849248a877a8abd5dd328, State: Initialized, Role: FOLLOWER
I20260923 22:21:05.951828 25080 consensus_queue.cc:260] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:05.951877 25081 raft_consensus.cc:348] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:05.952023 25081 raft_consensus.cc:374] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:05.952052 25081 raft_consensus.cc:729] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc39f5bfc9044094a62a9c5632e2f9c7, State: Initialized, Role: FOLLOWER
I20260923 22:21:05.952059 25080 ts_tablet_manager.cc:1428] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:05.952113 25065 heartbeater.cc:503] Master 127.4.207.62:43855 was elected leader, sending a full tablet report...
I20260923 22:21:05.952138 25081 consensus_queue.cc:260] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:05.952378 25081 ts_tablet_manager.cc:1428] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 22:21:05.952399 24803 heartbeater.cc:503] Master 127.4.207.62:43855 was elected leader, sending a full tablet report...
W20260923 22:21:06.046386 24935 tablet.cc:2369] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 22:21:06.047370 25086 raft_consensus.cc:482] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 22:21:06.047490 25086 raft_consensus.cc:504] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:06.047865 25086 leader_election.cc:288] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc39f5bfc9044094a62a9c5632e2f9c7 (127.4.207.1:42549), a631235c8c16432e961b688f859f0988 (127.4.207.2:38701)
I20260923 22:21:06.050976 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:06.048076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:38701 (local address 127.4.207.3:41961)
0923 22:21:06.048142 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:06.048154 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:06.048213 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:06.048424 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:06.048425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:06.048440 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:06.048520 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:06.048522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.049294 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.049296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:06.050132 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:06.050135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.050673 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.050675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:06.050748 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:06.050752 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:06.050808 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 22:21:06.050847 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":34}
I20260923 22:21:06.050979 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:06.048147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:41961 (local address 127.4.207.2:38701)
0923 22:21:06.048192 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:06.048194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:06.048205 (+ 11us) server_negotiation.cc:407] Connection header received
0923 22:21:06.048328 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:06.048329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:06.048340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:06.048402 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:06.048616 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.049198 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.050227 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.050588 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.050639 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:06.050646 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:06.050830 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 22:21:06.050879 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:06.051002 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:06.048076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:42549 (local address 127.4.207.3:34531)
0923 22:21:06.048278 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:06.048290 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:06.048366 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:06.048544 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:06.048545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:06.048547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 22:21:06.048652 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:06.048654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.049373 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.049374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:06.050151 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:06.050153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.050741 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.050741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:06.050819 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:06.050823 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 22:21:06.050877 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 22:21:06.050909 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":13,"thread_start_us":100,"threads_started":1}
I20260923 22:21:06.051023 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:06.048147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.207.3:34531 (local address 127.4.207.1:42549)
0923 22:21:06.048193 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:06.048195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:06.048372 (+ 177us) server_negotiation.cc:407] Connection header received
0923 22:21:06.048426 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:06.048428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:06.048443 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:06.048509 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 22:21:06.048767 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.049281 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.050271 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.050649 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.050725 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:06.050736 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:06.050896 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 22:21:06.050951 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:06.051363 24757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938318ed3db646a89d5b5544bde5c04c" candidate_uuid: "523cdabf532849248a877a8abd5dd328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" is_pre_election: true
I20260923 22:21:06.051396 24888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938318ed3db646a89d5b5544bde5c04c" candidate_uuid: "523cdabf532849248a877a8abd5dd328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a631235c8c16432e961b688f859f0988" is_pre_election: true
I20260923 22:21:06.051558 24757 raft_consensus.cc:2454] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 523cdabf532849248a877a8abd5dd328 in term 0.
I20260923 22:21:06.051577 24888 raft_consensus.cc:2454] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 523cdabf532849248a877a8abd5dd328 in term 0.
I20260923 22:21:06.051824 24954 leader_election.cc:302] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523cdabf532849248a877a8abd5dd328, dc39f5bfc9044094a62a9c5632e2f9c7; no voters:
I20260923 22:21:06.051965 25086 raft_consensus.cc:2781] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 22:21:06.052023 25086 raft_consensus.cc:482] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 22:21:06.052053 25086 raft_consensus.cc:3037] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:06.052798 25086 raft_consensus.cc:504] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:06.052950 25086 leader_election.cc:288] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [CANDIDATE]: Term 1 election: Requested vote from peers dc39f5bfc9044094a62a9c5632e2f9c7 (127.4.207.1:42549), a631235c8c16432e961b688f859f0988 (127.4.207.2:38701)
I20260923 22:21:06.053102 24757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938318ed3db646a89d5b5544bde5c04c" candidate_uuid: "523cdabf532849248a877a8abd5dd328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7"
I20260923 22:21:06.053107 24888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938318ed3db646a89d5b5544bde5c04c" candidate_uuid: "523cdabf532849248a877a8abd5dd328" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a631235c8c16432e961b688f859f0988"
I20260923 22:21:06.053188 24757 raft_consensus.cc:3037] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:06.053220 24888 raft_consensus.cc:3037] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:06.054030 24888 raft_consensus.cc:2454] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 523cdabf532849248a877a8abd5dd328 in term 1.
I20260923 22:21:06.054031 24757 raft_consensus.cc:2454] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 523cdabf532849248a877a8abd5dd328 in term 1.
I20260923 22:21:06.054205 24953 leader_election.cc:302] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523cdabf532849248a877a8abd5dd328, a631235c8c16432e961b688f859f0988; no voters:
I20260923 22:21:06.054306 25086 raft_consensus.cc:2781] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:06.054458 25086 raft_consensus.cc:686] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 1 LEADER]: Becoming Leader. State: Replica: 523cdabf532849248a877a8abd5dd328, State: Running, Role: LEADER
I20260923 22:21:06.054554 25086 consensus_queue.cc:237] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } }
I20260923 22:21:06.055344 24614 catalog_manager.cc:5495] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 reported cstate change: term changed from 0 to 1, leader changed from <none> to 523cdabf532849248a877a8abd5dd328 (127.4.207.3). New cstate: current_term: 1 leader_uuid: "523cdabf532849248a877a8abd5dd328" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "523cdabf532849248a877a8abd5dd328" member_type: VOTER last_known_addr { host: "127.4.207.3" port: 35553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 } health_report { overall_health: UNKNOWN } } }
I20260923 22:21:06.086639 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:06.084366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.4.207.3:35553)
0923 22:21:06.084406 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:06.084413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:06.084421 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:06.084516 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:06.084517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:06.084534 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:06.084587 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:06.084719 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.085162 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.085892 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:06.086084 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:06.086131 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:06.086323 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:06.086440 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:06.086512 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:06.086549 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 22:21:06.086566 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:06.088580 24888 raft_consensus.cc:1261] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988 [term 1 FOLLOWER]: Refusing update from remote peer 523cdabf532849248a877a8abd5dd328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:06.088581 24757 raft_consensus.cc:1261] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 1 FOLLOWER]: Refusing update from remote peer 523cdabf532849248a877a8abd5dd328: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:06.089020 25089 consensus_queue.cc:1035] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a631235c8c16432e961b688f859f0988" member_type: VOTER last_known_addr { host: "127.4.207.2" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:06.089169 25086 consensus_queue.cc:1035] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc39f5bfc9044094a62a9c5632e2f9c7" member_type: VOTER last_known_addr { host: "127.4.207.1" port: 42549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:06.091576 25092 mvcc.cc:205] Tried to move back new op lower bound from 7332667662695866368 to 7332667662560051200. Current Snapshot: MvccSnapshot[applied={T|T < 7332667662695866368}]
I20260923 22:21:06.092041 25094 mvcc.cc:205] Tried to move back new op lower bound from 7332667662695866368 to 7332667662560051200. Current Snapshot: MvccSnapshot[applied={T|T < 7332667662695866368}]
I20260923 22:21:06.092414 25093 mvcc.cc:205] Tried to move back new op lower bound from 7332667662695866368 to 7332667662560051200. Current Snapshot: MvccSnapshot[applied={T|T < 7332667662695866368 or (T in {7332667662695866368})}]
I20260923 22:21:06.093870 24614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:06.093964 24614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 22:21:06.094803 24614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 061446dda92f42cb93be5391d26c7b32: Sending DeleteTablet for 3 replicas of tablet 938318ed3db646a89d5b5544bde5c04c
I20260923 22:21:06.095121 24999 tablet_service.cc:1514] Processing DeleteTablet for tablet 938318ed3db646a89d5b5544bde5c04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:06 UTC) from {username='slave'} at 127.0.0.1:33046
I20260923 22:21:06.095132 24737 tablet_service.cc:1514] Processing DeleteTablet for tablet 938318ed3db646a89d5b5544bde5c04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:06 UTC) from {username='slave'} at 127.0.0.1:38366
I20260923 22:21:06.095206 24868 tablet_service.cc:1514] Processing DeleteTablet for tablet 938318ed3db646a89d5b5544bde5c04c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 22:21:06 UTC) from {username='slave'} at 127.0.0.1:36532
I20260923 22:21:06.095373 25103 tablet_replica.cc:330] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7: stopping tablet replica
I20260923 22:21:06.095410 25104 tablet_replica.cc:330] T 938318ed3db646a89d5b5544bde5c04c P a631235c8c16432e961b688f859f0988: stopping tablet replica
I20260923 22:21:06.095495 25103 raft_consensus.cc:2229] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 22:21:06.095638 25103 raft_consensus.cc:2258] T 938318ed3db646a89d5b5544bde5c04c P dc39f5bfc9044094a62a9c5632e2f9c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:06.095731 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 24675
I20260923 22:21:06.097203 25102 tablet_replica.cc:330] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: stopping tablet replica
I20260923 22:21:06.097337 25102 raft_consensus.cc:2229] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 1 LEADER]: Raft consensus shutting down.
I20260923 22:21:06.097437 25102 raft_consensus.cc:2258] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 22:21:06.097947 25102 ts_tablet_manager.cc:1905] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 22:21:06.099581 25102 ts_tablet_manager.cc:1918] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 22:21:06.099668 25102 log.cc:1196] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: Deleting WAL directory at /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790201827590652-4924-0/minicluster-data/ts-2/wal/wals/938318ed3db646a89d5b5544bde5c04c
I20260923 22:21:06.099925 25102 ts_tablet_manager.cc:1939] T 938318ed3db646a89d5b5544bde5c04c P 523cdabf532849248a877a8abd5dd328: Deleting consensus metadata
I20260923 22:21:06.100373 24600 catalog_manager.cc:4841] TS 523cdabf532849248a877a8abd5dd328 (127.4.207.3:35553): tablet 938318ed3db646a89d5b5544bde5c04c (table test-table [id=3a307c17ed6840e9a92c6907b8a20040]) successfully deleted
W20260923 22:21:06.101735 24601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.207.1:42549 (error 108)
W20260923 22:21:06.103359 25074 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:06.103187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.1:42549 (local address 127.0.0.1:38382)
0923 22:21:06.103242 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:06.103254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:06.103306 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.1:42549: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 22:21:06.103533 24601 catalog_manager.cc:4584] TS dc39f5bfc9044094a62a9c5632e2f9c7 (127.4.207.1:42549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 938318ed3db646a89d5b5544bde5c04c: Network error: Client connection negotiation failed: client connection to 127.4.207.1:42549: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260923 22:21:06.103605 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 24806
I20260923 22:21:06.110222 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 24937
I20260923 22:21:06.110332 25074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:06.110196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.2:38701 (local address 127.0.0.1:36538)
0923 22:21:06.110241 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:06.110298 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.2:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 22:21:06.114353 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 24584
2026-09-23T22:21:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (738 ms)
[----------] 6 tests from EncryptionPolicyTest (8633 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:06 dist-test-slave-7511 krb5kdc[25108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:06 dist-test-slave-7511 krb5kdc[25108](info): set up 2 sockets
Sep 23 22:21:06 dist-test-slave-7511 krb5kdc[25108](info): commencing operation
krb5kdc: starting...
W20260923 22:21:07.139261 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:07 dist-test-slave-7511 krb5kdc[25108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:07.258795 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:07.337100 25121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:07.337296 25121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:07.337316 25121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:07.338430 25121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:07.338474 25121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:07.338488 25121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:07.338501 25121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:07.338513 25121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:07.340032 25121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:07.340265 25121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:07.340481 25121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:07.345048 25129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:07.345011 25127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:07.345072 25126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:07.345156 25121 server_base.cc:1034] running on GCE node
I20260923 22:21:07.345486 25121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:07.345762 25121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:07.346917 25121 hybrid_clock.cc:648] HybridClock initialized: now 1790202067346910 us; error 40 us; skew 500 ppm
Sep 23 22:21:07 dist-test-slave-7511 krb5kdc[25108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:07.354521 25121 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:07.354875 25121 webserver.cc:460] Webserver started at http://127.0.0.1:33085/ using document root <none> and password file <none>
I20260923 22:21:07.355067 25121 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:07.355106 25121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:07.355187 25121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:07.356181 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "9921cc867dea41eebdeca48bd451a58b"
format_stamp: "Formatted at 2026-09-23 22:21:07 on dist-test-slave-7511"
server_key: "741c712e640f7fd423ba8f56fbbc47e3"
server_key_iv: "a7f93a099661730096b3a5649275da93"
server_key_version: "encryptionkey@0"
I20260923 22:21:07.356464 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "9921cc867dea41eebdeca48bd451a58b"
format_stamp: "Formatted at 2026-09-23 22:21:07 on dist-test-slave-7511"
server_key: "741c712e640f7fd423ba8f56fbbc47e3"
server_key_iv: "a7f93a099661730096b3a5649275da93"
server_key_version: "encryptionkey@0"
I20260923 22:21:07.357640 25121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 22:21:07.358289 25136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:07.358502 25121 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:07.358582 25121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "9921cc867dea41eebdeca48bd451a58b"
format_stamp: "Formatted at 2026-09-23 22:21:07 on dist-test-slave-7511"
server_key: "741c712e640f7fd423ba8f56fbbc47e3"
server_key_iv: "a7f93a099661730096b3a5649275da93"
server_key_version: "encryptionkey@0"
I20260923 22:21:07.358633 25121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:07.370306 25121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:07.372377 25121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:07.372483 25121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:07.376456 25121 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36323
I20260923 22:21:07.376523 25188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36323 every 8 connection(s)
I20260923 22:21:07.376880 25121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:07.377879 25189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:07.380153 25189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b: Bootstrap starting.
I20260923 22:21:07.380708 25189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:07.380986 25189 log.cc:824] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:07.381635 25189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b: No bootstrap required, opened a new log
I20260923 22:21:07.383159 25189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9921cc867dea41eebdeca48bd451a58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260923 22:21:07.383319 25189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:07.383407 25189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9921cc867dea41eebdeca48bd451a58b, State: Initialized, Role: FOLLOWER
I20260923 22:21:07.383602 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 25121
I20260923 22:21:07.383607 25189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9921cc867dea41eebdeca48bd451a58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260923 22:21:07.383688 25189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:07.383744 25189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:07.383750 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:07.383831 25189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:07.383988 4924 external_mini_cluster.cc:1428] Setting key 5e365b044e2555fe0990a57cd1966dc9
I20260923 22:21:07.384546 25189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9921cc867dea41eebdeca48bd451a58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260923 22:21:07.384653 25189 leader_election.cc:302] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9921cc867dea41eebdeca48bd451a58b; no voters:
I20260923 22:21:07.384799 25189 leader_election.cc:288] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:07.384853 25194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:07.385030 25189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:07.385040 25194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [term 1 LEADER]: Becoming Leader. State: Replica: 9921cc867dea41eebdeca48bd451a58b, State: Running, Role: LEADER
I20260923 22:21:07.385151 25194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9921cc867dea41eebdeca48bd451a58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260923 22:21:07.385452 25194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9921cc867dea41eebdeca48bd451a58b. Latest consensus state: current_term: 1 leader_uuid: "9921cc867dea41eebdeca48bd451a58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9921cc867dea41eebdeca48bd451a58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } } }
I20260923 22:21:07.385535 25194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:07.385704 25195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9921cc867dea41eebdeca48bd451a58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9921cc867dea41eebdeca48bd451a58b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } } }
I20260923 22:21:07.385828 25195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:07.385735 25200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:07.386627 25200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:07.388823 25200 catalog_manager.cc:1347] Generated new cluster ID: bd1e0867d2f14e709a7feab0e7dfd5c8
I20260923 22:21:07.388876 25200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:21:07 dist-test-slave-7511 krb5kdc[25108](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 22:21:07.395531 25200 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:07.395807 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:07.384460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60293 (local address 127.0.0.1:36323)
0923 22:21:07.384676 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:07.384685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:07.384718 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:21:07.384836 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:07.384843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:07.384910 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:07.385889 (+ 979us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:07.386008 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:07.387425 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:07.388474 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:07.389084 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:07.389229 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:07.391806 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:07.391810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:07.391810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:07.391837 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:07.394088 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:07.394529 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:07.394532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:07.394532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:07.394636 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:07.394887 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:07.394888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:07.394889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:07.395117 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:07.395306 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:07.395422 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 22:21:07.395568 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260923 22:21:07.396440 25200 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:07.396867 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:07.399662 25200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9921cc867dea41eebdeca48bd451a58b: Generated new TSK 0
I20260923 22:21:07.399821 25200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:07.406566 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:07.399522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38163 (local address 127.0.0.1:36323)
0923 22:21:07.399583 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:07.399584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:07.399931 (+ 347us) server_negotiation.cc:407] Connection header received
0923 22:21:07.399960 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:07.399961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:07.399973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:07.400069 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:07.400703 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:07.401196 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:07.402271 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:07.402486 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:07.402549 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:07.403949 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:07.403950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:07.403950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:07.403957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:07.405406 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:07.405846 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:07.405847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:07.405847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:07.405900 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:07.406150 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:07.406151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:07.406151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:07.406241 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:07.406316 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:07.406458 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 22:21:07.406510 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:21:07.408483 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 25121
W20260923 22:21:07.412398 25106 connection.cc:535] client connection to 127.0.0.1:36323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36323): Connection reset by peer (error 104)
2026-09-23T22:21:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:07 dist-test-slave-7511 krb5kdc[25220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:07 dist-test-slave-7511 krb5kdc[25220](info): set up 2 sockets
Sep 23 22:21:07 dist-test-slave-7511 krb5kdc[25220](info): commencing operation
krb5kdc: starting...
W20260923 22:21:08.435662 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:08 dist-test-slave-7511 krb5kdc[25220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:08.557662 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:08.639097 25233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:08.639348 25233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:08.639386 25233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:08.640574 25233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:08.640648 25233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:08.640674 25233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:08.640695 25233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:08.640725 25233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:08.642239 25233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39487
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:08.642527 25233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:08.642758 25233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:08.647630 25239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:08.647624 25241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:08.647629 25238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:08.647847 25233 server_base.cc:1034] running on GCE node
I20260923 22:21:08.648141 25233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:08.648463 25233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:08.649609 25233 hybrid_clock.cc:648] HybridClock initialized: now 1790202068649582 us; error 39 us; skew 500 ppm
Sep 23 22:21:08 dist-test-slave-7511 krb5kdc[25220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:08.652182 25233 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:08.652576 25233 webserver.cc:460] Webserver started at http://127.0.0.1:43821/ using document root <none> and password file <none>
I20260923 22:21:08.652815 25233 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:08.652863 25233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:08.653019 25233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:08.654098 25233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "04e7875186734e5b8f007bbfa497c9a3"
format_stamp: "Formatted at 2026-09-23 22:21:08 on dist-test-slave-7511"
server_key: "bce5947d780979638ad35120ef0162f2"
server_key_iv: "afecc664004eea6a03a172d49ae27c28"
server_key_version: "encryptionkey@0"
I20260923 22:21:08.654441 25233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "04e7875186734e5b8f007bbfa497c9a3"
format_stamp: "Formatted at 2026-09-23 22:21:08 on dist-test-slave-7511"
server_key: "bce5947d780979638ad35120ef0162f2"
server_key_iv: "afecc664004eea6a03a172d49ae27c28"
server_key_version: "encryptionkey@0"
I20260923 22:21:08.655844 25233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:08.656716 25248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:08.656896 25233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:08.656984 25233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "04e7875186734e5b8f007bbfa497c9a3"
format_stamp: "Formatted at 2026-09-23 22:21:08 on dist-test-slave-7511"
server_key: "bce5947d780979638ad35120ef0162f2"
server_key_iv: "afecc664004eea6a03a172d49ae27c28"
server_key_version: "encryptionkey@0"
I20260923 22:21:08.657040 25233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:08.682479 25233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:08.684475 25233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:08.684584 25233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:08.688395 25233 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39487
I20260923 22:21:08.688512 25300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39487 every 8 connection(s)
I20260923 22:21:08.688822 25233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:08.689704 25301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:08.692072 25301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3: Bootstrap starting.
I20260923 22:21:08.692574 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 25233
I20260923 22:21:08.692699 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:08.692814 25301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:08.692920 4924 external_mini_cluster.cc:1428] Setting key 96cfbe5752235349a0f97b0ac52b48d8
I20260923 22:21:08.693145 25301 log.cc:824] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:08.693857 25301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3: No bootstrap required, opened a new log
I20260923 22:21:08.695777 25301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e7875186734e5b8f007bbfa497c9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39487 } }
I20260923 22:21:08.695911 25301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:08.695933 25301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e7875186734e5b8f007bbfa497c9a3, State: Initialized, Role: FOLLOWER
I20260923 22:21:08.696064 25301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e7875186734e5b8f007bbfa497c9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39487 } }
I20260923 22:21:08.696135 25301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:08.696156 25301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:08.696216 25301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:08.696764 25301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e7875186734e5b8f007bbfa497c9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39487 } }
I20260923 22:21:08.696882 25301 leader_election.cc:302] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e7875186734e5b8f007bbfa497c9a3; no voters:
I20260923 22:21:08.697081 25301 leader_election.cc:288] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:08.697108 25306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:08.697352 25306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [term 1 LEADER]: Becoming Leader. State: Replica: 04e7875186734e5b8f007bbfa497c9a3, State: Running, Role: LEADER
I20260923 22:21:08.697397 25301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:08.697470 25306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e7875186734e5b8f007bbfa497c9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39487 } }
I20260923 22:21:08.697778 25306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e7875186734e5b8f007bbfa497c9a3. Latest consensus state: current_term: 1 leader_uuid: "04e7875186734e5b8f007bbfa497c9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e7875186734e5b8f007bbfa497c9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39487 } } }
I20260923 22:21:08.697801 25308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e7875186734e5b8f007bbfa497c9a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e7875186734e5b8f007bbfa497c9a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39487 } } }
I20260923 22:21:08.697928 25306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:08.698009 25308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:08.698426 25315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:08.698945 25315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:08.699715 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:08.693442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56363 (local address 127.0.0.1:39487)
0923 22:21:08.693615 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:08.693625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:08.693667 (+ 42us) server_negotiation.cc:407] Connection header received
0923 22:21:08.694159 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:08.694165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:08.694214 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:08.694356 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:08.694573 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:08.695501 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:08.696527 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:08.697009 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:08.697125 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:08.698910 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:08.698914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:08.698914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:08.699116 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:08.699172 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:08.699187 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:08.699264 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:08.699470 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 22:21:08.699498 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260923 22:21:08.700636 25315 catalog_manager.cc:1347] Generated new cluster ID: 650260c950c74df699be6e56e52a784b
I20260923 22:21:08.700711 25315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:08.700733 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:08.705577 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:08.702149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40415 (local address 127.0.0.1:39487)
0923 22:21:08.702190 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:08.702192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:08.702288 (+ 96us) server_negotiation.cc:407] Connection header received
0923 22:21:08.702371 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:08.702371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:08.702379 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:08.702404 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:08.702563 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:08.703263 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:08.704241 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:08.704547 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:08.704581 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:08.705335 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:08.705336 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:08.705336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:08.705340 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:08.705353 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:08.705357 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:08.705385 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:08.705499 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 22:21:08.705508 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 22:21:08.709944 25315 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:08.710651 25315 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:08.715374 25315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04e7875186734e5b8f007bbfa497c9a3: Generated new TSK 0
I20260923 22:21:08.715610 25315 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:08.717633 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 25233
W20260923 22:21:08.721926 25218 connection.cc:535] client connection to 127.0.0.1:39487 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39487): Connection reset by peer (error 104)
2026-09-23T22:21:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:08 dist-test-slave-7511 krb5kdc[25332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:08 dist-test-slave-7511 krb5kdc[25332](info): set up 2 sockets
Sep 23 22:21:08 dist-test-slave-7511 krb5kdc[25332](info): commencing operation
krb5kdc: starting...
W20260923 22:21:09.747393 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:09 dist-test-slave-7511 krb5kdc[25332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:09.862687 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:09.941249 25345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:09.941478 25345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:09.941512 25345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:09.942584 25345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:09.942646 25345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:09.942668 25345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:09.942695 25345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:09.942718 25345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:09.944090 25345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33543
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:09.944337 25345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:09.944545 25345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:09.949227 25353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:09.949232 25351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:09.949232 25350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:09.949393 25345 server_base.cc:1034] running on GCE node
I20260923 22:21:09.949697 25345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:09.949992 25345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:09.951189 25345 hybrid_clock.cc:648] HybridClock initialized: now 1790202069951184 us; error 48 us; skew 500 ppm
Sep 23 22:21:09 dist-test-slave-7511 krb5kdc[25332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:09.953609 25345 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:09.954011 25345 webserver.cc:460] Webserver started at http://127.0.0.1:39113/ using document root <none> and password file <none>
I20260923 22:21:09.954253 25345 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:09.954319 25345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:09.954463 25345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:09.955590 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "9dc592dfb92e4ab698b2c632e3bc6f90"
format_stamp: "Formatted at 2026-09-23 22:21:09 on dist-test-slave-7511"
server_key: "65db351090cd1e710b8ad42bb1021a06"
server_key_iv: "e203aec32ad5548def951e735cfa222f"
server_key_version: "encryptionkey@0"
I20260923 22:21:09.955937 25345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "9dc592dfb92e4ab698b2c632e3bc6f90"
format_stamp: "Formatted at 2026-09-23 22:21:09 on dist-test-slave-7511"
server_key: "65db351090cd1e710b8ad42bb1021a06"
server_key_iv: "e203aec32ad5548def951e735cfa222f"
server_key_version: "encryptionkey@0"
I20260923 22:21:09.957207 25345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:09.957906 25360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:09.958097 25345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:09.958204 25345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "9dc592dfb92e4ab698b2c632e3bc6f90"
format_stamp: "Formatted at 2026-09-23 22:21:09 on dist-test-slave-7511"
server_key: "65db351090cd1e710b8ad42bb1021a06"
server_key_iv: "e203aec32ad5548def951e735cfa222f"
server_key_version: "encryptionkey@0"
I20260923 22:21:09.958274 25345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:09.961472 25345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:09.963697 25345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:09.963841 25345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:09.967386 25345 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33543
I20260923 22:21:09.967413 25412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33543 every 8 connection(s)
I20260923 22:21:09.967758 25345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:09.968724 25413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:09.970886 25413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90: Bootstrap starting.
I20260923 22:21:09.971431 25413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:09.971786 25413 log.cc:824] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:09.972404 25413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90: No bootstrap required, opened a new log
I20260923 22:21:09.973883 25413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc592dfb92e4ab698b2c632e3bc6f90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } }
I20260923 22:21:09.974014 25413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:09.974036 25413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dc592dfb92e4ab698b2c632e3bc6f90, State: Initialized, Role: FOLLOWER
I20260923 22:21:09.974186 25413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc592dfb92e4ab698b2c632e3bc6f90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } }
I20260923 22:21:09.974246 25413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:09.974265 25413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:09.974313 25413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:09.974848 25413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc592dfb92e4ab698b2c632e3bc6f90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } }
I20260923 22:21:09.974932 25413 leader_election.cc:302] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dc592dfb92e4ab698b2c632e3bc6f90; no voters:
I20260923 22:21:09.975124 25413 leader_election.cc:288] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:09.975185 25416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:09.975435 25413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:09.975689 25416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [term 1 LEADER]: Becoming Leader. State: Replica: 9dc592dfb92e4ab698b2c632e3bc6f90, State: Running, Role: LEADER
I20260923 22:21:09.975826 25416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc592dfb92e4ab698b2c632e3bc6f90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } }
I20260923 22:21:09.976226 25417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dc592dfb92e4ab698b2c632e3bc6f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc592dfb92e4ab698b2c632e3bc6f90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } } }
I20260923 22:21:09.976233 25418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dc592dfb92e4ab698b2c632e3bc6f90. Latest consensus state: current_term: 1 leader_uuid: "9dc592dfb92e4ab698b2c632e3bc6f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dc592dfb92e4ab698b2c632e3bc6f90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33543 } } }
I20260923 22:21:09.976313 25417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:09.976336 25418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:09.976781 25425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:09.976831 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 25345
I20260923 22:21:09.976948 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:09.977154 4924 external_mini_cluster.cc:1428] Setting key 4ff11f3abae7345b21a0fe019b28302c
I20260923 22:21:09.977468 25425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:09.979831 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:09.977640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47817 (local address 127.0.0.1:33543)
0923 22:21:09.977810 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:09.977819 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:09.978245 (+ 426us) server_negotiation.cc:407] Connection header received
0923 22:21:09.978394 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:09.978419 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:09.978462 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:09.978617 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:09.979280 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:09.979284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:09.979284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:09.979288 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:09.979334 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:09.979345 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:09.979420 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:09.979569 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 22:21:09.979616 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260923 22:21:09.979882 25425 catalog_manager.cc:1347] Generated new cluster ID: f8dbf887da3749dcb430d5ca964f89bd
I20260923 22:21:09.979971 25425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:09.980840 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:09.984524 25435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:09.982269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50499 (local address 127.0.0.1:33543)
0923 22:21:09.982301 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:09.982302 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:09.983607 (+ 1305us) server_negotiation.cc:407] Connection header received
0923 22:21:09.983668 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:09.983669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:09.983674 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:09.983733 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:09.984291 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:09.984293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:09.984293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:09.984298 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:09.984317 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:09.984322 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:09.984384 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:09.984437 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 22:21:09.984452 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:09.991530 25425 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:09.991684 25425 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:10.001436 25425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dc592dfb92e4ab698b2c632e3bc6f90: Generated new TSK 0
I20260923 22:21:10.001618 25425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:10.005138 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 25345
2026-09-23T22:21:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:10 dist-test-slave-7511 krb5kdc[25444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:10 dist-test-slave-7511 krb5kdc[25444](info): set up 2 sockets
Sep 23 22:21:10 dist-test-slave-7511 krb5kdc[25444](info): commencing operation
krb5kdc: starting...
W20260923 22:21:11.019532 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:11 dist-test-slave-7511 krb5kdc[25444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:11.138253 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:11.217612 25457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:11.217860 25457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:11.217902 25457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:11.219094 25457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:11.219164 25457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:11.219189 25457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:11.219218 25457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:11.219249 25457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:11.220674 25457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:11.220938 25457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:11.221158 25457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:11.226063 25462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:11.226073 25465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:11.226075 25463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:11.226354 25457 server_base.cc:1034] running on GCE node
I20260923 22:21:11.226558 25457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:11.226783 25457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:11.227917 25457 hybrid_clock.cc:648] HybridClock initialized: now 1790202071227902 us; error 28 us; skew 500 ppm
Sep 23 22:21:11 dist-test-slave-7511 krb5kdc[25444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:11.230296 25457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:11.230741 25457 webserver.cc:460] Webserver started at http://127.0.0.1:36087/ using document root <none> and password file <none>
I20260923 22:21:11.230969 25457 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:11.231046 25457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:11.231196 25457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:11.232273 25457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61"
format_stamp: "Formatted at 2026-09-23 22:21:11 on dist-test-slave-7511"
server_key: "041de50a3387c0135fd58f0df639f6a2"
server_key_iv: "398e23752b82bf1eac9dc3b5c683a125"
server_key_version: "encryptionkey@0"
I20260923 22:21:11.232667 25457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61"
format_stamp: "Formatted at 2026-09-23 22:21:11 on dist-test-slave-7511"
server_key: "041de50a3387c0135fd58f0df639f6a2"
server_key_iv: "398e23752b82bf1eac9dc3b5c683a125"
server_key_version: "encryptionkey@0"
I20260923 22:21:11.233992 25457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:11.234766 25472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:11.234941 25457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:11.235065 25457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61"
format_stamp: "Formatted at 2026-09-23 22:21:11 on dist-test-slave-7511"
server_key: "041de50a3387c0135fd58f0df639f6a2"
server_key_iv: "398e23752b82bf1eac9dc3b5c683a125"
server_key_version: "encryptionkey@0"
I20260923 22:21:11.235144 25457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:11.245808 25457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:11.247694 25457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:11.247851 25457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:11.251626 25457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46155
I20260923 22:21:11.251653 25524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46155 every 8 connection(s)
I20260923 22:21:11.252005 25457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:11.252885 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 25457
I20260923 22:21:11.253012 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:11.253185 25525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:11.253245 4924 external_mini_cluster.cc:1428] Setting key 2e37cf2019adea3975ffa527dc13dc88
I20260923 22:21:11.256588 25525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61: Bootstrap starting.
I20260923 22:21:11.257246 25525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:11.257555 25525 log.cc:824] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:11.258206 25525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61: No bootstrap required, opened a new log
Sep 23 22:21:11 dist-test-slave-7511 krb5kdc[25444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 22:21:11.260036 25525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } }
I20260923 22:21:11.260165 25525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:11.260187 25525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c37a4b1b8c44f2ab3d4aaee68d98a61, State: Initialized, Role: FOLLOWER
I20260923 22:21:11.260330 25525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } }
I20260923 22:21:11.260407 25525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:11.260430 25525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:11.260509 25525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:11.261116 25525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } }
I20260923 22:21:11.261222 25525 leader_election.cc:302] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c37a4b1b8c44f2ab3d4aaee68d98a61; no voters:
I20260923 22:21:11.261395 25525 leader_election.cc:288] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:11.261425 25530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:11.261534 25530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [term 1 LEADER]: Becoming Leader. State: Replica: 2c37a4b1b8c44f2ab3d4aaee68d98a61, State: Running, Role: LEADER
I20260923 22:21:11.261656 25525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:11.261672 25530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } }
I20260923 22:21:11.262055 25530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c37a4b1b8c44f2ab3d4aaee68d98a61. Latest consensus state: current_term: 1 leader_uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } } }
I20260923 22:21:11.262321 25530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:11.262739 25539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:11.263046 25531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c37a4b1b8c44f2ab3d4aaee68d98a61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46155 } } }
I20260923 22:21:11.263192 25531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:11.263262 25539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:11.264269 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:11.253745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45533 (local address 127.0.0.1:46155)
0923 22:21:11.253917 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:11.253924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:11.253949 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:21:11.254360 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:11.254366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:11.254411 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:11.254523 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:11.255305 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:11.256198 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:11.257246 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:11.257712 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:11.257829 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:11.260186 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:11.260190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:11.260190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:11.260202 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:11.262170 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:11.262565 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:11.262567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:11.262568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:11.262682 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:11.262929 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:11.262930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:11.262931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:11.263236 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:11.263381 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:11.263842 (+ 461us) server_negotiation.cc:299] Negotiation successful
0923 22:21:11.263985 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260923 22:21:11.264989 25539 catalog_manager.cc:1347] Generated new cluster ID: 166bb205798a466b8230a9a71f6e58f4
I20260923 22:21:11.265036 25539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:11.271618 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:11.280439 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:11.273115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56825 (local address 127.0.0.1:46155)
0923 22:21:11.273173 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:11.273175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:11.273250 (+ 75us) server_negotiation.cc:407] Connection header received
0923 22:21:11.273311 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:11.273312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:11.273321 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:11.273383 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:11.274015 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:11.274741 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:11.275785 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:11.276136 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:11.276197 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:11.277633 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:11.277645 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:11.277645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:11.277651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:11.279234 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:11.279660 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:11.279661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:11.279661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:11.279725 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:11.279933 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:11.279934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:11.279934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:11.280056 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:11.280147 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:11.280274 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 22:21:11.280351 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 22:21:11.281548 25539 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:11.282037 25539 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:11.288403 25539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c37a4b1b8c44f2ab3d4aaee68d98a61: Generated new TSK 0
I20260923 22:21:11.288583 25539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:11.292675 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 25457
2026-09-23T22:21:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:11 dist-test-slave-7511 krb5kdc[25556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:11 dist-test-slave-7511 krb5kdc[25556](info): set up 2 sockets
Sep 23 22:21:11 dist-test-slave-7511 krb5kdc[25556](info): commencing operation
krb5kdc: starting...
W20260923 22:21:12.311657 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:12 dist-test-slave-7511 krb5kdc[25556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:12.430680 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:12.512255 25569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:12.512495 25569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:12.512533 25569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:12.513674 25569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:12.513739 25569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:12.513765 25569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:12.513795 25569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:12.513820 25569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:12.515282 25569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:12.515573 25569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:12.515800 25569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:12.520641 25577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:12.520681 25569 server_base.cc:1034] running on GCE node
W20260923 22:21:12.520715 25574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:12.520634 25575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:12.521204 25569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:12.521441 25569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:12.522578 25569 hybrid_clock.cc:648] HybridClock initialized: now 1790202072522559 us; error 44 us; skew 500 ppm
Sep 23 22:21:12 dist-test-slave-7511 krb5kdc[25556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:12.529234 25569 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:12.529716 25569 webserver.cc:460] Webserver started at http://127.0.0.1:41385/ using document root <none> and password file <none>
I20260923 22:21:12.529965 25569 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:12.530030 25569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:12.530193 25569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:12.531203 25569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "cf27f60b693742bb8979179be5f262de"
format_stamp: "Formatted at 2026-09-23 22:21:12 on dist-test-slave-7511"
server_key: "7b575459d27592b3ed298924664653ac"
server_key_iv: "187fd078d438b8784cb2d05370948879"
server_key_version: "encryptionkey@0"
I20260923 22:21:12.531567 25569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "cf27f60b693742bb8979179be5f262de"
format_stamp: "Formatted at 2026-09-23 22:21:12 on dist-test-slave-7511"
server_key: "7b575459d27592b3ed298924664653ac"
server_key_iv: "187fd078d438b8784cb2d05370948879"
server_key_version: "encryptionkey@0"
I20260923 22:21:12.532874 25569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:12.533612 25584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:12.533772 25569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:12.533893 25569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "cf27f60b693742bb8979179be5f262de"
format_stamp: "Formatted at 2026-09-23 22:21:12 on dist-test-slave-7511"
server_key: "7b575459d27592b3ed298924664653ac"
server_key_iv: "187fd078d438b8784cb2d05370948879"
server_key_version: "encryptionkey@0"
I20260923 22:21:12.533967 25569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:12.554809 25569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:12.557207 25569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:12.557376 25569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:12.561411 25569 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41977
I20260923 22:21:12.561450 25636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41977 every 8 connection(s)
I20260923 22:21:12.561793 25569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:12.562707 25637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:12.564817 25637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de: Bootstrap starting.
I20260923 22:21:12.565347 25637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:12.565528 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 25569
I20260923 22:21:12.565660 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:12.565696 25637 log.cc:824] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:12.565868 4924 external_mini_cluster.cc:1428] Setting key 517d7e73f85fb899c703a30e4c6c7986
I20260923 22:21:12.566538 25637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de: No bootstrap required, opened a new log
I20260923 22:21:12.569041 25637 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27f60b693742bb8979179be5f262de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } }
I20260923 22:21:12.569201 25637 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:12.569231 25637 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf27f60b693742bb8979179be5f262de, State: Initialized, Role: FOLLOWER
I20260923 22:21:12.569341 25637 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27f60b693742bb8979179be5f262de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } }
I20260923 22:21:12.569410 25637 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:12.569440 25637 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:12.569497 25637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:12.570147 25637 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27f60b693742bb8979179be5f262de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } }
I20260923 22:21:12.570256 25637 leader_election.cc:302] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf27f60b693742bb8979179be5f262de; no voters:
I20260923 22:21:12.570407 25637 leader_election.cc:288] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:12.570425 25642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:12.570545 25642 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [term 1 LEADER]: Becoming Leader. State: Replica: cf27f60b693742bb8979179be5f262de, State: Running, Role: LEADER
I20260923 22:21:12.570632 25642 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27f60b693742bb8979179be5f262de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } }
I20260923 22:21:12.570668 25637 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:12.570947 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf27f60b693742bb8979179be5f262de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27f60b693742bb8979179be5f262de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } } }
I20260923 22:21:12.571213 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:12.570945 25642 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf27f60b693742bb8979179be5f262de. Latest consensus state: current_term: 1 leader_uuid: "cf27f60b693742bb8979179be5f262de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf27f60b693742bb8979179be5f262de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41977 } } }
I20260923 22:21:12.571683 25642 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:12.571733 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:12.566393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44697 (local address 127.0.0.1:41977)
0923 22:21:12.566575 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:12.566584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:12.566616 (+ 32us) server_negotiation.cc:407] Connection header received
0923 22:21:12.567013 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:12.567022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:12.567071 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:12.567178 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:12.567409 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:12.568344 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:12.569530 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:12.570168 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:12.570285 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:12.570960 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:12.570964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:12.570964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:12.571173 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:12.571211 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:12.571226 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:12.571269 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:12.571415 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 22:21:12.571531 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260923 22:21:12.571822 25651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:12.572324 25651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:12.573155 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:12.573717 25651 catalog_manager.cc:1347] Generated new cluster ID: 09fa5bd6184940bc9005c3cd3e805148
I20260923 22:21:12.573765 25651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:12.578295 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:12.574677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47449 (local address 127.0.0.1:41977)
0923 22:21:12.574716 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:12.574717 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:12.575609 (+ 892us) server_negotiation.cc:407] Connection header received
0923 22:21:12.575680 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:12.575681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:12.575691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:12.575732 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:12.575879 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:12.576393 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:12.577125 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:12.577327 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:12.577408 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:12.578052 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:12.578053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:12.578054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:12.578058 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:12.578071 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:12.578075 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:12.578127 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:12.578185 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 22:21:12.578204 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:12.584048 25651 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:12.584725 25651 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:12.594652 25651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf27f60b693742bb8979179be5f262de: Generated new TSK 0
I20260923 22:21:12.594851 25651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:12.601195 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 25569
2026-09-23T22:21:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:12 dist-test-slave-7511 krb5kdc[25668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:12 dist-test-slave-7511 krb5kdc[25668](info): set up 2 sockets
Sep 23 22:21:12 dist-test-slave-7511 krb5kdc[25668](info): commencing operation
krb5kdc: starting...
W20260923 22:21:13.611713 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:13 dist-test-slave-7511 krb5kdc[25668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:13.726745 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:13.805325 25681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:13.805567 25681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:13.805616 25681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:13.806720 25681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:13.806788 25681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:13.806813 25681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:13.806841 25681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:13.806867 25681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:13.808341 25681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:13.808593 25681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:13.808853 25681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:13.813884 25687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:13.813905 25686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:13.813896 25689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:13.814059 25681 server_base.cc:1034] running on GCE node
I20260923 22:21:13.814266 25681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:13.814491 25681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:13.815625 25681 hybrid_clock.cc:648] HybridClock initialized: now 1790202073815615 us; error 32 us; skew 500 ppm
Sep 23 22:21:13 dist-test-slave-7511 krb5kdc[25668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:13.818017 25681 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:13.818463 25681 webserver.cc:460] Webserver started at http://127.0.0.1:39731/ using document root <none> and password file <none>
I20260923 22:21:13.818657 25681 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:13.818697 25681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:13.818779 25681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:13.819794 25681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "b43ed81b3d7f4acbb6fc237a4956de1e"
format_stamp: "Formatted at 2026-09-23 22:21:13 on dist-test-slave-7511"
server_key: "dfda77475614637e6829a173b60e9095"
server_key_iv: "8c1663718e76c87d38879b8a94805d45"
server_key_version: "encryptionkey@0"
I20260923 22:21:13.820088 25681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "b43ed81b3d7f4acbb6fc237a4956de1e"
format_stamp: "Formatted at 2026-09-23 22:21:13 on dist-test-slave-7511"
server_key: "dfda77475614637e6829a173b60e9095"
server_key_iv: "8c1663718e76c87d38879b8a94805d45"
server_key_version: "encryptionkey@0"
I20260923 22:21:13.821466 25681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:13.822209 25696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:13.822381 25681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:13.822469 25681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "b43ed81b3d7f4acbb6fc237a4956de1e"
format_stamp: "Formatted at 2026-09-23 22:21:13 on dist-test-slave-7511"
server_key: "dfda77475614637e6829a173b60e9095"
server_key_iv: "8c1663718e76c87d38879b8a94805d45"
server_key_version: "encryptionkey@0"
I20260923 22:21:13.822544 25681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:13.825780 25681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:13.827914 25681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:13.828066 25681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:13.832259 25681 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37193
I20260923 22:21:13.832345 25748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37193 every 8 connection(s)
I20260923 22:21:13.832647 25681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:13.833750 25749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:13.836272 25749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e: Bootstrap starting.
I20260923 22:21:13.836889 25749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:13.837193 25749 log.cc:824] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:13.837896 25749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e: No bootstrap required, opened a new log
I20260923 22:21:13.839681 25749 raft_consensus.cc:348] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43ed81b3d7f4acbb6fc237a4956de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260923 22:21:13.839844 25749 raft_consensus.cc:374] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:13.839897 25749 raft_consensus.cc:729] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43ed81b3d7f4acbb6fc237a4956de1e, State: Initialized, Role: FOLLOWER
I20260923 22:21:13.840072 25749 consensus_queue.cc:260] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43ed81b3d7f4acbb6fc237a4956de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260923 22:21:13.840173 25749 raft_consensus.cc:388] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:13.840216 25749 raft_consensus.cc:482] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:13.840268 25749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:13.840840 25749 raft_consensus.cc:504] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43ed81b3d7f4acbb6fc237a4956de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260923 22:21:13.840873 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 25681
I20260923 22:21:13.840960 25749 leader_election.cc:302] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b43ed81b3d7f4acbb6fc237a4956de1e; no voters:
I20260923 22:21:13.840982 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:13.841172 25749 leader_election.cc:288] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:13.841235 25752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:13.841260 4924 external_mini_cluster.cc:1428] Setting key f5f05d6d7c3e495442038b599c24babf
I20260923 22:21:13.841423 25752 raft_consensus.cc:686] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [term 1 LEADER]: Becoming Leader. State: Replica: b43ed81b3d7f4acbb6fc237a4956de1e, State: Running, Role: LEADER
I20260923 22:21:13.841449 25749 sys_catalog.cc:564] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:13.842780 25752 consensus_queue.cc:237] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43ed81b3d7f4acbb6fc237a4956de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260923 22:21:13.843559 25753 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b43ed81b3d7f4acbb6fc237a4956de1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43ed81b3d7f4acbb6fc237a4956de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } } }
I20260923 22:21:13.843648 25753 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:13.844034 25754 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b43ed81b3d7f4acbb6fc237a4956de1e. Latest consensus state: current_term: 1 leader_uuid: "b43ed81b3d7f4acbb6fc237a4956de1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b43ed81b3d7f4acbb6fc237a4956de1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } } }
I20260923 22:21:13.844090 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:13.841900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57693 (local address 127.0.0.1:37193)
0923 22:21:13.842123 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:13.842135 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:13.842172 (+ 37us) server_negotiation.cc:407] Connection header received
0923 22:21:13.842289 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:13.842297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:13.842327 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:13.842502 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:13.843594 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:13.843598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:13.843598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:13.843603 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:13.843649 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:13.843665 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:13.843714 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:13.843849 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 22:21:13.843892 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260923 22:21:13.844213 25754 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:13.844201 25769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:13.844926 25769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:13.845129 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:13.846406 25769 catalog_manager.cc:1347] Generated new cluster ID: 0566d7e9704949a6b99c64140c3eb9de
I20260923 22:21:13.846458 25769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:13.847888 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:13.846832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52785 (local address 127.0.0.1:37193)
0923 22:21:13.846859 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:13.846861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:13.846892 (+ 31us) server_negotiation.cc:407] Connection header received
0923 22:21:13.846969 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:13.846970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:13.846974 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:13.847037 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:13.847696 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:13.847697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:13.847698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:13.847701 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:13.847712 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:13.847716 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:13.847741 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:13.847824 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 22:21:13.847835 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260923 22:21:13.860347 25769 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:13.860514 25769 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:13.867705 25769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b43ed81b3d7f4acbb6fc237a4956de1e: Generated new TSK 0
I20260923 22:21:13.867893 25769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:13.875708 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 25681
2026-09-23T22:21:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:13 dist-test-slave-7511 krb5kdc[25780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:13 dist-test-slave-7511 krb5kdc[25780](info): set up 2 sockets
Sep 23 22:21:13 dist-test-slave-7511 krb5kdc[25780](info): commencing operation
krb5kdc: starting...
W20260923 22:21:14.975668 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:15 dist-test-slave-7511 krb5kdc[25780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:15.094283 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:15.174741 25793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:15.174973 25793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:15.175007 25793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:15.176182 25793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:15.176250 25793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:15.176275 25793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:15.176308 25793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:15.176337 25793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:15.177765 25793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:15.178071 25793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:15.178292 25793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:15.183220 25801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:15.183234 25799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:15.183389 25793 server_base.cc:1034] running on GCE node
W20260923 22:21:15.183491 25798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:15.183758 25793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:15.184083 25793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:15.185252 25793 hybrid_clock.cc:648] HybridClock initialized: now 1790202075185231 us; error 32 us; skew 500 ppm
Sep 23 22:21:15 dist-test-slave-7511 krb5kdc[25780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:15.187634 25793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:15.188035 25793 webserver.cc:460] Webserver started at http://127.0.0.1:37477/ using document root <none> and password file <none>
I20260923 22:21:15.188303 25793 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:15.188371 25793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:15.188535 25793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:15.189591 25793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "af77363918964e77a8ef7b586b339cfe"
format_stamp: "Formatted at 2026-09-23 22:21:15 on dist-test-slave-7511"
server_key: "b9f648106296b271de012a1be40c7fb5"
server_key_iv: "6374540ef7f7480dd09c9f4eacca2a41"
server_key_version: "encryptionkey@0"
I20260923 22:21:15.189942 25793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "af77363918964e77a8ef7b586b339cfe"
format_stamp: "Formatted at 2026-09-23 22:21:15 on dist-test-slave-7511"
server_key: "b9f648106296b271de012a1be40c7fb5"
server_key_iv: "6374540ef7f7480dd09c9f4eacca2a41"
server_key_version: "encryptionkey@0"
I20260923 22:21:15.191228 25793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:15.191987 25808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:15.192158 25793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:15.192251 25793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "af77363918964e77a8ef7b586b339cfe"
format_stamp: "Formatted at 2026-09-23 22:21:15 on dist-test-slave-7511"
server_key: "b9f648106296b271de012a1be40c7fb5"
server_key_iv: "6374540ef7f7480dd09c9f4eacca2a41"
server_key_version: "encryptionkey@0"
I20260923 22:21:15.192337 25793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:15.205492 25793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:15.207413 25793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:15.207617 25793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:15.211337 25793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44087
I20260923 22:21:15.211375 25860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44087 every 8 connection(s)
I20260923 22:21:15.211747 25793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:15.212702 25861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:15.214879 25861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe: Bootstrap starting.
I20260923 22:21:15.215392 25861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:15.215700 25861 log.cc:824] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:15.216346 25861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe: No bootstrap required, opened a new log
I20260923 22:21:15.217842 25861 raft_consensus.cc:348] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af77363918964e77a8ef7b586b339cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44087 } }
I20260923 22:21:15.217974 25861 raft_consensus.cc:374] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:15.217996 25861 raft_consensus.cc:729] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af77363918964e77a8ef7b586b339cfe, State: Initialized, Role: FOLLOWER
I20260923 22:21:15.218137 25861 consensus_queue.cc:260] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af77363918964e77a8ef7b586b339cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44087 } }
I20260923 22:21:15.218219 25861 raft_consensus.cc:388] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:15.218242 25861 raft_consensus.cc:482] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:15.218271 25861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:15.218736 25861 raft_consensus.cc:504] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af77363918964e77a8ef7b586b339cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44087 } }
I20260923 22:21:15.218818 25861 leader_election.cc:302] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af77363918964e77a8ef7b586b339cfe; no voters:
I20260923 22:21:15.218961 25861 leader_election.cc:288] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:15.219053 25864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:15.219132 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 25793
I20260923 22:21:15.219226 25861 sys_catalog.cc:564] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:15.219262 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:15.219506 4924 external_mini_cluster.cc:1428] Setting key 93dc623a48bc985bf42b0031ce26559f
I20260923 22:21:15.219219 25864 raft_consensus.cc:686] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [term 1 LEADER]: Becoming Leader. State: Replica: af77363918964e77a8ef7b586b339cfe, State: Running, Role: LEADER
I20260923 22:21:15.219956 25864 consensus_queue.cc:237] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af77363918964e77a8ef7b586b339cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44087 } }
I20260923 22:21:15.220590 25865 sys_catalog.cc:455] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af77363918964e77a8ef7b586b339cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af77363918964e77a8ef7b586b339cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44087 } } }
I20260923 22:21:15.220768 25865 sys_catalog.cc:458] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:15.220641 25866 sys_catalog.cc:455] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader af77363918964e77a8ef7b586b339cfe. Latest consensus state: current_term: 1 leader_uuid: "af77363918964e77a8ef7b586b339cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af77363918964e77a8ef7b586b339cfe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44087 } } }
I20260923 22:21:15.221174 25866 sys_catalog.cc:458] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:15.221184 25880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:15.221346 25880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:15.221426 25883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:15.221585 25883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:15.223127 25883 catalog_manager.cc:1347] Generated new cluster ID: 4556d59ba0764487977fd6e2fbbc9cfd
I20260923 22:21:15.223177 25883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:21:15 dist-test-slave-7511 krb5kdc[25780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 22:21:15.231101 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:15.220358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:56267 (local address 127.0.0.1:44087)
0923 22:21:15.220512 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:15.220520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:15.220553 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:21:15.221140 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:15.221150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:15.221216 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:15.221369 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:15.222126 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:15.223539 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:15.224406 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:15.224850 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:15.224990 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:15.227575 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:15.227582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:15.227582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:15.227599 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:15.229403 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:15.229838 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:15.229843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:15.229844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:15.229958 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:15.230234 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:15.230236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:15.230236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:15.230492 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:15.230685 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:15.230777 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 22:21:15.230885 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260923 22:21:15.232101 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:15.237751 25883 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:15.238646 25883 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:15.240563 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:15.233504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:59121 (local address 127.0.0.1:44087)
0923 22:21:15.233546 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:15.233548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:15.233699 (+ 151us) server_negotiation.cc:407] Connection header received
0923 22:21:15.233775 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:15.233776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:15.233787 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:15.233844 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:15.234485 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:15.235011 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:15.235917 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:15.236157 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:15.236246 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:15.237612 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:15.237614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:15.237614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:15.237620 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:15.239264 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:15.239729 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:15.239730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:15.239731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:15.239790 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:15.240022 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:15.240023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:15.240023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:15.240157 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:15.240261 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:15.240414 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 22:21:15.240476 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:15.249392 25883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af77363918964e77a8ef7b586b339cfe: Generated new TSK 0
I20260923 22:21:15.249650 25883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:15.253959 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 25793
W20260923 22:21:15.258235 25778 connection.cc:535] client connection to 127.0.0.1:44087 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44087): Connection reset by peer (error 104)
2026-09-23T22:21:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:15 dist-test-slave-7511 krb5kdc[25892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:15 dist-test-slave-7511 krb5kdc[25892](info): set up 2 sockets
Sep 23 22:21:15 dist-test-slave-7511 krb5kdc[25892](info): commencing operation
krb5kdc: starting...
W20260923 22:21:16.263571 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:16 dist-test-slave-7511 krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:16.376080 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:16.455807 25905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:16.456034 25905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:16.456074 25905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:16.457175 25905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:16.457244 25905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:16.457271 25905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:16.457302 25905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:16.457336 25905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:16.458724 25905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:16.458986 25905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:16.459252 25905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:21:16.464303 25905 server_base.cc:1034] running on GCE node
W20260923 22:21:16.464383 25913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:16.464352 25910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:16.464483 25911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:16.464740 25905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:16.464985 25905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:16.466126 25905 hybrid_clock.cc:648] HybridClock initialized: now 1790202076466109 us; error 41 us; skew 500 ppm
Sep 23 22:21:16 dist-test-slave-7511 krb5kdc[25892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:16.468511 25905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:16.468922 25905 webserver.cc:460] Webserver started at http://127.0.0.1:42239/ using document root <none> and password file <none>
I20260923 22:21:16.469111 25905 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:16.469151 25905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:16.469237 25905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:16.470203 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "3e80b53b2c524bba92bedc39f7263171"
format_stamp: "Formatted at 2026-09-23 22:21:16 on dist-test-slave-7511"
server_key: "6a2c5c2d52f97002851f31fa910df490"
server_key_iv: "2de677c32bdafa750c1d6fb6f023f316"
server_key_version: "encryptionkey@0"
I20260923 22:21:16.470494 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "3e80b53b2c524bba92bedc39f7263171"
format_stamp: "Formatted at 2026-09-23 22:21:16 on dist-test-slave-7511"
server_key: "6a2c5c2d52f97002851f31fa910df490"
server_key_iv: "2de677c32bdafa750c1d6fb6f023f316"
server_key_version: "encryptionkey@0"
I20260923 22:21:16.471800 25905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:16.472464 25920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:16.472651 25905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:16.472743 25905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "3e80b53b2c524bba92bedc39f7263171"
format_stamp: "Formatted at 2026-09-23 22:21:16 on dist-test-slave-7511"
server_key: "6a2c5c2d52f97002851f31fa910df490"
server_key_iv: "2de677c32bdafa750c1d6fb6f023f316"
server_key_version: "encryptionkey@0"
I20260923 22:21:16.472823 25905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:16.482777 25905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:16.484577 25905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:16.484725 25905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:16.488775 25905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44401
I20260923 22:21:16.488801 25972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44401 every 8 connection(s)
I20260923 22:21:16.489125 25905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:16.490072 25973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:16.490397 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 25905
I20260923 22:21:16.490643 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:16.490832 4924 external_mini_cluster.cc:1428] Setting key 4006760778d35a28af351bd0bb27deba
I20260923 22:21:16.493338 25973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171: Bootstrap starting.
I20260923 22:21:16.494585 25973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:16.494899 25973 log.cc:824] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:16.495903 25973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171: No bootstrap required, opened a new log
I20260923 22:21:16.497929 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:16.491333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:57203 (local address 127.0.0.1:44401)
0923 22:21:16.491536 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:16.491546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:16.491580 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:16.492108 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:16.492116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:16.492177 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:16.492324 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:16.492489 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:16.493920 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:16.495073 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:16.495672 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:16.495773 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:16.496753 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:16.496758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:16.496758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:16.496965 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:16.497031 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:16.497097 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:16.497255 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:16.497424 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 22:21:16.497451 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":89,"threads_started":1}
I20260923 22:21:16.498163 25973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e80b53b2c524bba92bedc39f7263171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } }
I20260923 22:21:16.498350 25973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:16.498396 25973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e80b53b2c524bba92bedc39f7263171, State: Initialized, Role: FOLLOWER
I20260923 22:21:16.498515 25973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e80b53b2c524bba92bedc39f7263171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } }
I20260923 22:21:16.498610 25973 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:16.498641 25973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:16.498714 25973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:16.499480 25973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e80b53b2c524bba92bedc39f7263171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } }
I20260923 22:21:16.499595 25973 leader_election.cc:302] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e80b53b2c524bba92bedc39f7263171; no voters:
I20260923 22:21:16.499696 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:16.499794 25973 leader_election.cc:288] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:16.499869 25978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:16.500046 25973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:16.500052 25978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [term 1 LEADER]: Becoming Leader. State: Replica: 3e80b53b2c524bba92bedc39f7263171, State: Running, Role: LEADER
I20260923 22:21:16.500262 25978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e80b53b2c524bba92bedc39f7263171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } }
I20260923 22:21:16.500635 25978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e80b53b2c524bba92bedc39f7263171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e80b53b2c524bba92bedc39f7263171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } } }
I20260923 22:21:16.500770 25978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:16.500928 25978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e80b53b2c524bba92bedc39f7263171. Latest consensus state: current_term: 1 leader_uuid: "3e80b53b2c524bba92bedc39f7263171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e80b53b2c524bba92bedc39f7263171" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44401 } } }
I20260923 22:21:16.501042 25978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171 [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:16.501784 25997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:16.501871 25997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:16.502128 25987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:16.502307 25987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:16.503968 25987 catalog_manager.cc:1347] Generated new cluster ID: 2cee0f2ed34a4e18a9d9027d275cd453
I20260923 22:21:16.504011 25987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:16.507650 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:16.503318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:58769 (local address 127.0.0.1:44401)
0923 22:21:16.503352 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:16.503354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:16.503688 (+ 334us) server_negotiation.cc:407] Connection header received
0923 22:21:16.503785 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:16.503786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:16.503795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:16.503823 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:16.504207 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:16.504855 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:16.506001 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:16.506301 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:16.506343 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:16.507342 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:16.507343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:16.507343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:16.507348 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:16.507361 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:16.507366 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:16.507399 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:16.507556 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 22:21:16.507578 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:16.509457 25987 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:16.510149 25987 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:16.523250 25987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e80b53b2c524bba92bedc39f7263171: Generated new TSK 0
I20260923 22:21:16.523488 25987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:16.526702 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 25905
W20260923 22:21:16.536633 25890 connection.cc:535] client connection to 127.0.0.1:44401 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44401): Connection reset by peer (error 104)
2026-09-23T22:21:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:16 dist-test-slave-7511 krb5kdc[26005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:16 dist-test-slave-7511 krb5kdc[26005](info): set up 2 sockets
Sep 23 22:21:16 dist-test-slave-7511 krb5kdc[26005](info): commencing operation
krb5kdc: starting...
W20260923 22:21:17.579703 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:17 dist-test-slave-7511 krb5kdc[26005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:17.699543 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:17.778877 26018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:17.779157 26018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:17.779199 26018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:17.780385 26018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:17.780462 26018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:17.780486 26018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:17.780508 26018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:17.780537 26018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:17.782011 26018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:17.782294 26018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:17.782507 26018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:17.787107 26026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:17.787147 26023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:17.787147 26024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:17.787305 26018 server_base.cc:1034] running on GCE node
I20260923 22:21:17.787566 26018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:17.787860 26018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:17.789023 26018 hybrid_clock.cc:648] HybridClock initialized: now 1790202077789010 us; error 42 us; skew 500 ppm
Sep 23 22:21:17 dist-test-slave-7511 krb5kdc[26005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:17.795109 26018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:17.795629 26018 webserver.cc:460] Webserver started at http://127.0.0.1:35065/ using document root <none> and password file <none>
I20260923 22:21:17.795859 26018 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:17.795925 26018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:17.796056 26018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:17.797124 26018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "0865b8d17bd64c668c3e1782d14a53fb"
format_stamp: "Formatted at 2026-09-23 22:21:17 on dist-test-slave-7511"
server_key: "eb73fef902e66036ac09882d809b7e47"
server_key_iv: "43ad4131a2fe678bfff4a65b4a7a91e7"
server_key_version: "encryptionkey@0"
I20260923 22:21:17.797492 26018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "0865b8d17bd64c668c3e1782d14a53fb"
format_stamp: "Formatted at 2026-09-23 22:21:17 on dist-test-slave-7511"
server_key: "eb73fef902e66036ac09882d809b7e47"
server_key_iv: "43ad4131a2fe678bfff4a65b4a7a91e7"
server_key_version: "encryptionkey@0"
I20260923 22:21:17.798880 26018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:17.799823 26033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:17.800019 26018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:17.800143 26018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "0865b8d17bd64c668c3e1782d14a53fb"
format_stamp: "Formatted at 2026-09-23 22:21:17 on dist-test-slave-7511"
server_key: "eb73fef902e66036ac09882d809b7e47"
server_key_iv: "43ad4131a2fe678bfff4a65b4a7a91e7"
server_key_version: "encryptionkey@0"
I20260923 22:21:17.800230 26018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:17.803521 26018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:17.805274 26018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:17.805434 26018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:17.809317 26018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45505
I20260923 22:21:17.809347 26085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45505 every 8 connection(s)
I20260923 22:21:17.809657 26018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:17.810631 26086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:17.813179 26086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb: Bootstrap starting.
I20260923 22:21:17.813704 26086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:17.813934 26086 log.cc:824] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:17.814145 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26018
I20260923 22:21:17.814283 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:17.814468 4924 external_mini_cluster.cc:1428] Setting key c159d4d328cc4a1c8623a207aab1546d
I20260923 22:21:17.814760 26086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb: No bootstrap required, opened a new log
I20260923 22:21:17.816977 26086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0865b8d17bd64c668c3e1782d14a53fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45505 } }
I20260923 22:21:17.817160 26086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:17.817201 26086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0865b8d17bd64c668c3e1782d14a53fb, State: Initialized, Role: FOLLOWER
I20260923 22:21:17.817343 26086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0865b8d17bd64c668c3e1782d14a53fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45505 } }
I20260923 22:21:17.817478 26086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:17.817524 26086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:17.817585 26086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:17.817824 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:17.815016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:58851 (local address 127.0.0.1:45505)
0923 22:21:17.815216 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:17.815227 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:17.815262 (+ 35us) server_negotiation.cc:407] Connection header received
0923 22:21:17.816138 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:17.816147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:17.816181 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:17.816339 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:17.817143 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:17.817147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:17.817147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:17.817152 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:17.817199 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:17.817214 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:17.817283 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:17.817394 (+ 111us) server_negotiation.cc:299] Negotiation successful
0923 22:21:17.817438 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260923 22:21:17.818320 26086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0865b8d17bd64c668c3e1782d14a53fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45505 } }
I20260923 22:21:17.818446 26086 leader_election.cc:302] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0865b8d17bd64c668c3e1782d14a53fb; no voters:
I20260923 22:21:17.818645 26086 leader_election.cc:288] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:17.818675 26091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:17.818815 26091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [term 1 LEADER]: Becoming Leader. State: Replica: 0865b8d17bd64c668c3e1782d14a53fb, State: Running, Role: LEADER
I20260923 22:21:17.818852 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:17.818895 26091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0865b8d17bd64c668c3e1782d14a53fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45505 } }
I20260923 22:21:17.818938 26086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:17.819204 26091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0865b8d17bd64c668c3e1782d14a53fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0865b8d17bd64c668c3e1782d14a53fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45505 } } }
I20260923 22:21:17.819326 26091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:17.819515 26091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0865b8d17bd64c668c3e1782d14a53fb. Latest consensus state: current_term: 1 leader_uuid: "0865b8d17bd64c668c3e1782d14a53fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0865b8d17bd64c668c3e1782d14a53fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45505 } } }
I20260923 22:21:17.819653 26091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:17.820824 26111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:17.820883 26111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:17.820937 26096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:17.821130 26096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:17.821586 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:17.820422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:37503 (local address 127.0.0.1:45505)
0923 22:21:17.820454 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:17.820456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:17.820592 (+ 136us) server_negotiation.cc:407] Connection header received
0923 22:21:17.820707 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:17.820708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:17.820712 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:17.820800 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:17.821379 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:17.821380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:17.821380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:17.821385 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:17.821397 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:17.821401 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:17.821457 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:17.821512 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 22:21:17.821526 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:21:17.822633 26096 catalog_manager.cc:1347] Generated new cluster ID: 9850f0db64bd415a90d54a4079e429bd
I20260923 22:21:17.822693 26096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:17.835139 26096 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:17.835325 26096 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:17.847564 26096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0865b8d17bd64c668c3e1782d14a53fb: Generated new TSK 0
I20260923 22:21:17.847776 26096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:17.856287 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26018
2026-09-23T22:21:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:17 dist-test-slave-7511 krb5kdc[26117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:17 dist-test-slave-7511 krb5kdc[26117](info): set up 2 sockets
Sep 23 22:21:17 dist-test-slave-7511 krb5kdc[26117](info): commencing operation
krb5kdc: starting...
W20260923 22:21:18.875576 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:18 dist-test-slave-7511 krb5kdc[26117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:18.992535 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:19.073738 26130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:19.073940 26130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:19.073962 26130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:19.075094 26130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:19.075135 26130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:19.075150 26130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:19.075162 26130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:19.075212 26130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:19.076690 26130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37507
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:19.076927 26130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:19.077157 26130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:19.081909 26135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:19.081981 26138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:19.081935 26136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:19.082068 26130 server_base.cc:1034] running on GCE node
I20260923 22:21:19.082284 26130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:19.082569 26130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:19.083716 26130 hybrid_clock.cc:648] HybridClock initialized: now 1790202079083698 us; error 43 us; skew 500 ppm
Sep 23 22:21:19 dist-test-slave-7511 krb5kdc[26117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:19.086148 26130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:19.086614 26130 webserver.cc:460] Webserver started at http://127.0.0.1:36987/ using document root <none> and password file <none>
I20260923 22:21:19.086834 26130 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:19.086911 26130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:19.087047 26130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:19.088111 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "feb28efdb71f4ecbb6988baf5f758a70"
format_stamp: "Formatted at 2026-09-23 22:21:19 on dist-test-slave-7511"
server_key: "ff19132b0f6ba03aed5648c176b56335"
server_key_iv: "e804dce41ead26c95a2e3266664027e3"
server_key_version: "encryptionkey@0"
I20260923 22:21:19.088459 26130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "feb28efdb71f4ecbb6988baf5f758a70"
format_stamp: "Formatted at 2026-09-23 22:21:19 on dist-test-slave-7511"
server_key: "ff19132b0f6ba03aed5648c176b56335"
server_key_iv: "e804dce41ead26c95a2e3266664027e3"
server_key_version: "encryptionkey@0"
I20260923 22:21:19.089759 26130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:19.090456 26145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:19.090623 26130 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:19.090731 26130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "feb28efdb71f4ecbb6988baf5f758a70"
format_stamp: "Formatted at 2026-09-23 22:21:19 on dist-test-slave-7511"
server_key: "ff19132b0f6ba03aed5648c176b56335"
server_key_iv: "e804dce41ead26c95a2e3266664027e3"
server_key_version: "encryptionkey@0"
I20260923 22:21:19.090821 26130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:19.099195 26130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:19.101370 26130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:19.101539 26130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:19.105520 26130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37507
I20260923 22:21:19.105546 26197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37507 every 8 connection(s)
I20260923 22:21:19.105883 26130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:19.106624 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26130
I20260923 22:21:19.106745 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:19.106823 26198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:19.106947 4924 external_mini_cluster.cc:1428] Setting key d533390125418a10c77c62eb5c9f491f
I20260923 22:21:19.109467 26198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70: Bootstrap starting.
I20260923 22:21:19.110050 26198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:19.110320 26198 log.cc:824] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:19.110978 26198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70: No bootstrap required, opened a new log
I20260923 22:21:19.113397 26198 raft_consensus.cc:348] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb28efdb71f4ecbb6988baf5f758a70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37507 } }
I20260923 22:21:19.113539 26198 raft_consensus.cc:374] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 23 22:21:19 dist-test-slave-7511 krb5kdc[26117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 22:21:19.113582 26198 raft_consensus.cc:729] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb28efdb71f4ecbb6988baf5f758a70, State: Initialized, Role: FOLLOWER
I20260923 22:21:19.113705 26198 consensus_queue.cc:260] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb28efdb71f4ecbb6988baf5f758a70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37507 } }
I20260923 22:21:19.113788 26198 raft_consensus.cc:388] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:19.113827 26198 raft_consensus.cc:482] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:19.113889 26198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:19.114370 26198 raft_consensus.cc:504] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb28efdb71f4ecbb6988baf5f758a70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37507 } }
I20260923 22:21:19.114498 26198 leader_election.cc:302] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: feb28efdb71f4ecbb6988baf5f758a70; no voters:
I20260923 22:21:19.114663 26198 leader_election.cc:288] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:19.114696 26203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:19.114830 26203 raft_consensus.cc:686] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [term 1 LEADER]: Becoming Leader. State: Replica: feb28efdb71f4ecbb6988baf5f758a70, State: Running, Role: LEADER
I20260923 22:21:19.114959 26198 sys_catalog.cc:564] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:19.115013 26203 consensus_queue.cc:237] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb28efdb71f4ecbb6988baf5f758a70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37507 } }
I20260923 22:21:19.115532 26204 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader feb28efdb71f4ecbb6988baf5f758a70. Latest consensus state: current_term: 1 leader_uuid: "feb28efdb71f4ecbb6988baf5f758a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb28efdb71f4ecbb6988baf5f758a70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37507 } } }
I20260923 22:21:19.115540 26205 sys_catalog.cc:455] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "feb28efdb71f4ecbb6988baf5f758a70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "feb28efdb71f4ecbb6988baf5f758a70" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37507 } } }
I20260923 22:21:19.115697 26204 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:19.115732 26205 sys_catalog.cc:458] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:19.115909 26212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:19.116420 26212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:19.118307 26212 catalog_manager.cc:1347] Generated new cluster ID: 1988b1d84efd4c08bd3b2e9689cdef72
I20260923 22:21:19.118363 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:19.107523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:38655 (local address 127.0.0.1:37507)
0923 22:21:19.107709 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:19.107717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:19.107741 (+ 24us) server_negotiation.cc:407] Connection header received
0923 22:21:19.108164 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:19.108170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:19.108215 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:19.108358 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:19.108976 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:19.109929 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:19.110932 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:19.111397 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:19.111535 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:19.114151 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:19.114155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:19.114156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:19.114166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:19.116434 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:19.117052 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:19.117055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:19.117055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:19.117157 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:19.117409 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:19.117410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:19.117411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:19.117729 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:19.117895 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:19.118054 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 22:21:19.118158 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260923 22:21:19.118371 26212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:19.119294 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:19.130728 26201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:19.123782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:41871 (local address 127.0.0.1:37507)
0923 22:21:19.123815 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:19.123817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:19.123877 (+ 60us) server_negotiation.cc:407] Connection header received
0923 22:21:19.123901 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:19.123901 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:19.123910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:19.123982 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:19.124500 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:19.125235 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:19.126159 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:19.126489 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:19.126556 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:19.127910 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:19.127911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:19.127912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:19.127919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:19.129523 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:19.129913 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:19.129914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:19.129915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:19.129975 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:19.130194 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:19.130195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:19.130195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:19.130322 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:19.130424 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:19.130554 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 22:21:19.130629 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:19.133823 26212 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:19.134506 26212 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:19.137750 26212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P feb28efdb71f4ecbb6988baf5f758a70: Generated new TSK 0
I20260923 22:21:19.137913 26212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:19.139669 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26130
W20260923 22:21:19.143973 26115 connection.cc:535] client connection to 127.0.0.1:37507 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37507): Connection reset by peer (error 104)
2026-09-23T22:21:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:19 dist-test-slave-7511 krb5kdc[26229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:19 dist-test-slave-7511 krb5kdc[26229](info): set up 2 sockets
Sep 23 22:21:19 dist-test-slave-7511 krb5kdc[26229](info): commencing operation
krb5kdc: starting...
W20260923 22:21:20.143393 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:20 dist-test-slave-7511 krb5kdc[26229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:20.260859 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:20.343254 26242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:20.343518 26242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:20.343567 26242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:20.344722 26242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:20.344766 26242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:20.344782 26242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:20.344796 26242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:20.344834 26242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:20.346336 26242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:20.346614 26242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:20.346863 26242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:20.351778 26247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:20.351900 26242 server_base.cc:1034] running on GCE node
W20260923 22:21:20.351778 26248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:20.351786 26250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:20.352226 26242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:20.352555 26242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:20.353726 26242 hybrid_clock.cc:648] HybridClock initialized: now 1790202080353722 us; error 35 us; skew 500 ppm
Sep 23 22:21:20 dist-test-slave-7511 krb5kdc[26229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:20.356112 26242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:20.356562 26242 webserver.cc:460] Webserver started at http://127.0.0.1:46227/ using document root <none> and password file <none>
I20260923 22:21:20.356817 26242 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:20.356884 26242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:20.357029 26242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:20.358093 26242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "3e4267b7e04c47e19ac0b231f85dd6e7"
format_stamp: "Formatted at 2026-09-23 22:21:20 on dist-test-slave-7511"
server_key: "bb38875fd67495724d87efe2b6ea1dea"
server_key_iv: "94e7e49ad8e183e6e8225b32b4cccd8c"
server_key_version: "encryptionkey@0"
I20260923 22:21:20.358453 26242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "3e4267b7e04c47e19ac0b231f85dd6e7"
format_stamp: "Formatted at 2026-09-23 22:21:20 on dist-test-slave-7511"
server_key: "bb38875fd67495724d87efe2b6ea1dea"
server_key_iv: "94e7e49ad8e183e6e8225b32b4cccd8c"
server_key_version: "encryptionkey@0"
I20260923 22:21:20.359923 26242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:20.360644 26257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:20.360847 26242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:20.360940 26242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "3e4267b7e04c47e19ac0b231f85dd6e7"
format_stamp: "Formatted at 2026-09-23 22:21:20 on dist-test-slave-7511"
server_key: "bb38875fd67495724d87efe2b6ea1dea"
server_key_iv: "94e7e49ad8e183e6e8225b32b4cccd8c"
server_key_version: "encryptionkey@0"
I20260923 22:21:20.360991 26242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:20.371980 26242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:20.373886 26242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:20.373996 26242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:20.377981 26242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37149
I20260923 22:21:20.378316 26309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37149 every 8 connection(s)
I20260923 22:21:20.378372 26242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:20.379107 26310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:20.381582 26310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7: Bootstrap starting.
I20260923 22:21:20.382256 26310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:20.382555 26310 log.cc:824] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:20.383279 26310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7: No bootstrap required, opened a new log
I20260923 22:21:20.384950 26310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4267b7e04c47e19ac0b231f85dd6e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260923 22:21:20.385113 26310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:20.385201 26310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4267b7e04c47e19ac0b231f85dd6e7, State: Initialized, Role: FOLLOWER
I20260923 22:21:20.385375 26310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4267b7e04c47e19ac0b231f85dd6e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260923 22:21:20.385458 26310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:20.385504 26310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:20.385558 26310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:20.385886 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26242
I20260923 22:21:20.386026 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:20.386271 4924 external_mini_cluster.cc:1428] Setting key 9112ad75fc5ebf5867adc5c89cc037c0
I20260923 22:21:20.386601 26310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4267b7e04c47e19ac0b231f85dd6e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260923 22:21:20.386716 26310 leader_election.cc:302] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e4267b7e04c47e19ac0b231f85dd6e7; no voters:
I20260923 22:21:20.386960 26310 leader_election.cc:288] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:20.387105 26315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:20.387233 26310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:20.387259 26315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [term 1 LEADER]: Becoming Leader. State: Replica: 3e4267b7e04c47e19ac0b231f85dd6e7, State: Running, Role: LEADER
I20260923 22:21:20.387542 26315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4267b7e04c47e19ac0b231f85dd6e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } }
I20260923 22:21:20.387925 26317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e4267b7e04c47e19ac0b231f85dd6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4267b7e04c47e19ac0b231f85dd6e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } } }
I20260923 22:21:20.388046 26317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:20.387917 26316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4267b7e04c47e19ac0b231f85dd6e7. Latest consensus state: current_term: 1 leader_uuid: "3e4267b7e04c47e19ac0b231f85dd6e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4267b7e04c47e19ac0b231f85dd6e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37149 } } }
I20260923 22:21:20.388176 26316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7 [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:20.389134 26330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:20.389202 26330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:20.389286 26324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:20.389451 26324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:20.391165 26324 catalog_manager.cc:1347] Generated new cluster ID: bc38faf036a54074be3beb31035f1a6a
I20260923 22:21:20.391270 26324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:20.392663 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:20.386805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:32823 (local address 127.0.0.1:37149)
0923 22:21:20.387013 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:20.387023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:20.387072 (+ 49us) server_negotiation.cc:407] Connection header received
0923 22:21:20.387200 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:20.387207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:20.387264 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:20.387469 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:20.387729 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:20.389312 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:20.390427 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:20.391075 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:20.391222 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:20.391983 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:20.391987 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:20.391988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:20.392174 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:20.392207 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:20.392217 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:20.392319 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:20.392403 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 22:21:20.392440 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260923 22:21:20.393582 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:20.397900 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:20.394929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:56371 (local address 127.0.0.1:37149)
0923 22:21:20.394966 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:20.394968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:20.395102 (+ 134us) server_negotiation.cc:407] Connection header received
0923 22:21:20.395178 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:20.395179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:20.395190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:20.395252 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:20.395399 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:20.395887 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:20.396606 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:20.396832 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:20.396922 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:20.397634 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:20.397636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:20.397636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:20.397640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:20.397656 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:20.397660 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:20.397718 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:20.397792 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 22:21:20.397805 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:20.420065 26324 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:20.420840 26324 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:20.427577 26324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e4267b7e04c47e19ac0b231f85dd6e7: Generated new TSK 0
I20260923 22:21:20.427791 26324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:20.430022 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26242
W20260923 22:21:20.433887 26227 connection.cc:535] client connection to 127.0.0.1:37149 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37149): Connection reset by peer (error 104)
2026-09-23T22:21:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:20 dist-test-slave-7511 krb5kdc[26341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:20 dist-test-slave-7511 krb5kdc[26341](info): set up 2 sockets
Sep 23 22:21:20 dist-test-slave-7511 krb5kdc[26341](info): commencing operation
krb5kdc: starting...
W20260923 22:21:21.439429 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:21 dist-test-slave-7511 krb5kdc[26341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 22:21:21.553994 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38331
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:21.634119 26354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:21.634332 26354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:21.634356 26354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:21.635560 26354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:21.635601 26354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:21.635615 26354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:21.635628 26354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:21.635669 26354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:21.637135 26354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38331
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:21.637370 26354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:21.637584 26354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:21.642449 26359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:21.642462 26360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:21.642465 26362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:21.642706 26354 server_base.cc:1034] running on GCE node
I20260923 22:21:21.642974 26354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:21.643275 26354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:21.644425 26354 hybrid_clock.cc:648] HybridClock initialized: now 1790202081644402 us; error 38 us; skew 500 ppm
Sep 23 22:21:21 dist-test-slave-7511 krb5kdc[26341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:21.646898 26354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 22:21:21.647316 26354 webserver.cc:460] Webserver started at http://127.0.0.1:40273/ using document root <none> and password file <none>
I20260923 22:21:21.647593 26354 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:21.647661 26354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:21.647800 26354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:21.648819 26354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "f7d6b4e78e164126a94868045b844eb1"
format_stamp: "Formatted at 2026-09-23 22:21:21 on dist-test-slave-7511"
server_key: "1edd55a52774ed61e09ca2e5723db757"
server_key_iv: "900de48b0ef872635486d24063813dff"
server_key_version: "encryptionkey@0"
I20260923 22:21:21.649216 26354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "f7d6b4e78e164126a94868045b844eb1"
format_stamp: "Formatted at 2026-09-23 22:21:21 on dist-test-slave-7511"
server_key: "1edd55a52774ed61e09ca2e5723db757"
server_key_iv: "900de48b0ef872635486d24063813dff"
server_key_version: "encryptionkey@0"
I20260923 22:21:21.650686 26354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:21.651391 26369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:21.651587 26354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:21.651690 26354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "f7d6b4e78e164126a94868045b844eb1"
format_stamp: "Formatted at 2026-09-23 22:21:21 on dist-test-slave-7511"
server_key: "1edd55a52774ed61e09ca2e5723db757"
server_key_iv: "900de48b0ef872635486d24063813dff"
server_key_version: "encryptionkey@0"
I20260923 22:21:21.651777 26354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:21.654867 26354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:21.656817 26354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:21.656981 26354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:21.661130 26354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38331
I20260923 22:21:21.661167 26421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38331 every 8 connection(s)
I20260923 22:21:21.661526 26354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:21.662581 26422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:21.664832 26422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1: Bootstrap starting.
I20260923 22:21:21.665354 26422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:21.665616 26422 log.cc:824] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:21.666293 26422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1: No bootstrap required, opened a new log
I20260923 22:21:21.668094 26422 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d6b4e78e164126a94868045b844eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260923 22:21:21.668229 26422 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:21.668251 26422 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7d6b4e78e164126a94868045b844eb1, State: Initialized, Role: FOLLOWER
I20260923 22:21:21.668381 26422 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d6b4e78e164126a94868045b844eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260923 22:21:21.668457 26422 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:21.668479 26422 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:21.668507 26422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:21.668991 26422 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d6b4e78e164126a94868045b844eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260923 22:21:21.669013 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26354
I20260923 22:21:21.669080 26422 leader_election.cc:302] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7d6b4e78e164126a94868045b844eb1; no voters:
I20260923 22:21:21.669145 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:21.669286 26422 leader_election.cc:288] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:21.669348 26425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:21.669390 4924 external_mini_cluster.cc:1428] Setting key 34f77f8f0d5ec74bcab688cf58179d7d
I20260923 22:21:21.669509 26425 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [term 1 LEADER]: Becoming Leader. State: Replica: f7d6b4e78e164126a94868045b844eb1, State: Running, Role: LEADER
I20260923 22:21:21.669596 26422 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:21.669679 26425 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d6b4e78e164126a94868045b844eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } }
I20260923 22:21:21.670153 26425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7d6b4e78e164126a94868045b844eb1. Latest consensus state: current_term: 1 leader_uuid: "f7d6b4e78e164126a94868045b844eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d6b4e78e164126a94868045b844eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } } }
I20260923 22:21:21.670159 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7d6b4e78e164126a94868045b844eb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7d6b4e78e164126a94868045b844eb1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38331 } } }
I20260923 22:21:21.670310 26425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:21.670401 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:21.670881 26433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:21.671381 26433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:21.672207 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:21.669899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:54617 (local address 127.0.0.1:38331)
0923 22:21:21.670296 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:21.670476 (+ 180us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:21.670544 (+ 68us) server_negotiation.cc:407] Connection header received
0923 22:21:21.670678 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:21.670687 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:21.670743 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:21.670912 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:21.671661 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:21.671664 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:21.671665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:21.671669 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:21.671712 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:21.671726 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:21.671793 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:21.671973 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 22:21:21.672024 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":62,"threads_started":1}
I20260923 22:21:21.673173 26433 catalog_manager.cc:1347] Generated new cluster ID: 7733e345eed84ba899b3a706e77a7d55
I20260923 22:21:21.673223 26433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:21.675505 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:21.685170 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:21.683772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:50315 (local address 127.0.0.1:38331)
0923 22:21:21.683815 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:21.683817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:21.684088 (+ 271us) server_negotiation.cc:407] Connection header received
0923 22:21:21.684216 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:21.684217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:21.684224 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:21.684275 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:21.684942 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:21.684943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:21.684944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:21.684948 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:21.684963 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:21.684967 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:21.685013 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:21.685067 (+ 54us) server_negotiation.cc:299] Negotiation successful
0923 22:21:21.685083 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:21.687719 26433 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:21.687887 26433 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:21.692641 26433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7d6b4e78e164126a94868045b844eb1: Generated new TSK 0
I20260923 22:21:21.692809 26433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:21.696031 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26354
2026-09-23T22:21:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1266 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:21 dist-test-slave-7511 krb5kdc[26453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:21 dist-test-slave-7511 krb5kdc[26453](info): set up 2 sockets
Sep 23 22:21:21 dist-test-slave-7511 krb5kdc[26453](info): commencing operation
krb5kdc: starting...
W20260923 22:21:22.707499 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:22 dist-test-slave-7511 krb5kdc[26453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:22.824523 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:45093
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:36407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:45093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:22.902860 26466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:22.903072 26466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:22.903095 26466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:22.904265 26466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:22.904307 26466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:22.904321 26466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:22.904335 26466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:22.904346 26466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:22.905803 26466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:36407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:45093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:22.906090 26466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:22.906318 26466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:22.911059 26472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:22.911105 26466 server_base.cc:1034] running on GCE node
W20260923 22:21:22.911159 26471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:22.911038 26474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:22.911556 26466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:22.911808 26466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:22.912961 26466 hybrid_clock.cc:648] HybridClock initialized: now 1790202082912946 us; error 28 us; skew 500 ppm
Sep 23 22:21:22 dist-test-slave-7511 krb5kdc[26453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202082, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:22.918701 26466 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:22.919198 26466 webserver.cc:460] Webserver started at http://127.4.207.62:43627/ using document root <none> and password file <none>
I20260923 22:21:22.919492 26466 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:22.919561 26466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:22.919698 26466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:22.920706 26466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "ec0c76b71b3d4d0c81829e97b64cc82b"
format_stamp: "Formatted at 2026-09-23 22:21:22 on dist-test-slave-7511"
server_key: "0e8b1832c088362af7cc5341789632c0"
server_key_iv: "08353e19714b1651e3b7c936a62ee34f"
server_key_version: "encryptionkey@0"
I20260923 22:21:22.921005 26466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "ec0c76b71b3d4d0c81829e97b64cc82b"
format_stamp: "Formatted at 2026-09-23 22:21:22 on dist-test-slave-7511"
server_key: "0e8b1832c088362af7cc5341789632c0"
server_key_iv: "08353e19714b1651e3b7c936a62ee34f"
server_key_version: "encryptionkey@0"
I20260923 22:21:22.922250 26466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 22:21:22.922977 26481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:22.923223 26466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:22.923295 26466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "ec0c76b71b3d4d0c81829e97b64cc82b"
format_stamp: "Formatted at 2026-09-23 22:21:22 on dist-test-slave-7511"
server_key: "0e8b1832c088362af7cc5341789632c0"
server_key_iv: "08353e19714b1651e3b7c936a62ee34f"
server_key_version: "encryptionkey@0"
I20260923 22:21:22.923344 26466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:22.931985 26466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:22.933643 26466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:22.933745 26466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:22.937738 26466 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:45093
I20260923 22:21:22.938050 26533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:45093 every 8 connection(s)
I20260923 22:21:22.938099 26466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:22.938678 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26466
I20260923 22:21:22.938705 26534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:22.938805 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:22.938985 4924 external_mini_cluster.cc:1428] Setting key 24a13218eaa21c00dde6796b52bc18ea
I20260923 22:21:22.941860 26534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b: Bootstrap starting.
I20260923 22:21:22.942597 26534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:22.942898 26534 log.cc:824] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:22.943675 26534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b: No bootstrap required, opened a new log
Sep 23 22:21:22 dist-test-slave-7511 krb5kdc[26453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:22.946173 26534 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0c76b71b3d4d0c81829e97b64cc82b" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45093 } }
I20260923 22:21:22.946337 26534 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:22.946388 26534 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec0c76b71b3d4d0c81829e97b64cc82b, State: Initialized, Role: FOLLOWER
I20260923 22:21:22.946568 26534 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0c76b71b3d4d0c81829e97b64cc82b" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45093 } }
I20260923 22:21:22.946658 26534 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:22.946715 26534 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:22.946767 26534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:22.947256 26534 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0c76b71b3d4d0c81829e97b64cc82b" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45093 } }
I20260923 22:21:22.947372 26534 leader_election.cc:302] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec0c76b71b3d4d0c81829e97b64cc82b; no voters:
I20260923 22:21:22.947557 26534 leader_election.cc:288] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:22.947590 26539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:22.947698 26539 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [term 1 LEADER]: Becoming Leader. State: Replica: ec0c76b71b3d4d0c81829e97b64cc82b, State: Running, Role: LEADER
I20260923 22:21:22.947808 26534 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:22.947860 26539 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0c76b71b3d4d0c81829e97b64cc82b" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45093 } }
I20260923 22:21:22.948194 26539 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec0c76b71b3d4d0c81829e97b64cc82b. Latest consensus state: current_term: 1 leader_uuid: "ec0c76b71b3d4d0c81829e97b64cc82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0c76b71b3d4d0c81829e97b64cc82b" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45093 } } }
I20260923 22:21:22.948200 26541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec0c76b71b3d4d0c81829e97b64cc82b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec0c76b71b3d4d0c81829e97b64cc82b" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 45093 } } }
I20260923 22:21:22.948355 26539 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:22.948376 26541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:22.948719 26548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:22.949280 26548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:22.950246 26537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:22.939556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51077 (local address 127.4.207.62:45093)
0923 22:21:22.939729 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:22.939737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:22.939761 (+ 24us) server_negotiation.cc:407] Connection header received
0923 22:21:22.940197 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:22.940203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:22.940248 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:22.940359 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:22.941152 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:22.942062 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:22.943195 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:22.943778 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:22.943875 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:22.946376 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:22.946380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:22.946380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:22.946392 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:22.948494 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:22.948957 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:22.948959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:22.948960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:22.949068 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:22.949338 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:22.949339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:22.949339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:22.949653 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:22.949819 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:22.949934 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 22:21:22.950032 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260923 22:21:22.951478 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:22.952523 26548 catalog_manager.cc:1347] Generated new cluster ID: 9c80aef3df704beea0592d20324ecf58
I20260923 22:21:22.952594 26548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:22.963235 26537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:22.955833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36541 (local address 127.4.207.62:45093)
0923 22:21:22.955868 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:22.955870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:22.956027 (+ 157us) server_negotiation.cc:407] Connection header received
0923 22:21:22.956055 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:22.956056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:22.956067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:22.956130 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:22.956721 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:22.957475 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:22.958422 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:22.958760 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:22.958833 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:22.960206 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:22.960207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:22.960208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:22.960215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:22.961864 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:22.962269 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:22.962270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:22.962270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:22.962331 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:22.962684 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:22.962685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:22.962685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:22.962830 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:22.962932 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:22.963073 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 22:21:22.963141 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:22.993988 26548 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:22.994583 26548 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:22.997700 26548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec0c76b71b3d4d0c81829e97b64cc82b: Generated new TSK 0
I20260923 22:21:22.997922 26548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:23.000036 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26466
W20260923 22:21:23.004334 26451 connection.cc:535] client connection to 127.4.207.62:45093 recv error: Network error: failed to read from TLS socket (remote: 127.4.207.62:45093): Connection reset by peer (error 104)
2026-09-23T22:21:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:23 dist-test-slave-7511 krb5kdc[26565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:23 dist-test-slave-7511 krb5kdc[26565](info): set up 2 sockets
Sep 23 22:21:23 dist-test-slave-7511 krb5kdc[26565](info): commencing operation
krb5kdc: starting...
W20260923 22:21:24.007601 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:24 dist-test-slave-7511 krb5kdc[26565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:24.127341 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39997
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:39737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:39997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:24.205394 26578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:24.205595 26578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:24.205618 26578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:24.206812 26578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:24.206861 26578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:24.206877 26578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:24.206945 26578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:24.206974 26578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:24.208461 26578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:39737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:39997
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:39997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:24.208721 26578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:24.209012 26578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:24.214184 26583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:24.214221 26586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:24.214111 26584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:24.214331 26578 server_base.cc:1034] running on GCE node
I20260923 22:21:24.214723 26578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:24.215039 26578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:24.216194 26578 hybrid_clock.cc:648] HybridClock initialized: now 1790202084216189 us; error 34 us; skew 500 ppm
Sep 23 22:21:24 dist-test-slave-7511 krb5kdc[26565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202084, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:24.218477 26578 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:24.218931 26578 webserver.cc:460] Webserver started at http://127.4.207.62:33789/ using document root <none> and password file <none>
I20260923 22:21:24.219178 26578 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:24.219242 26578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:24.219373 26578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:24.220386 26578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "1a270bb4b27942abaf964ecfd39565ee"
format_stamp: "Formatted at 2026-09-23 22:21:24 on dist-test-slave-7511"
server_key: "b4a5cf10a1ecebbe1791db67dcbc635a"
server_key_iv: "163525eedfdeb1deaf4e280aacefec09"
server_key_version: "encryptionkey@0"
I20260923 22:21:24.220746 26578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "1a270bb4b27942abaf964ecfd39565ee"
format_stamp: "Formatted at 2026-09-23 22:21:24 on dist-test-slave-7511"
server_key: "b4a5cf10a1ecebbe1791db67dcbc635a"
server_key_iv: "163525eedfdeb1deaf4e280aacefec09"
server_key_version: "encryptionkey@0"
I20260923 22:21:24.222059 26578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:24.222779 26593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:24.222954 26578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:24.223055 26578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "1a270bb4b27942abaf964ecfd39565ee"
format_stamp: "Formatted at 2026-09-23 22:21:24 on dist-test-slave-7511"
server_key: "b4a5cf10a1ecebbe1791db67dcbc635a"
server_key_iv: "163525eedfdeb1deaf4e280aacefec09"
server_key_version: "encryptionkey@0"
I20260923 22:21:24.223133 26578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:24.232990 26578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:24.234673 26578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:24.234828 26578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:24.238835 26578 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:39997
I20260923 22:21:24.238868 26645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:39997 every 8 connection(s)
I20260923 22:21:24.239190 26578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:24.240054 26646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:24.241669 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26578
I20260923 22:21:24.241804 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:24.241987 4924 external_mini_cluster.cc:1428] Setting key 9e8fe53a8bc6c1943dbbf14df6964970
I20260923 22:21:24.242470 26646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee: Bootstrap starting.
I20260923 22:21:24.243046 26646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:24.243369 26646 log.cc:824] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:24.244145 26646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee: No bootstrap required, opened a new log
I20260923 22:21:24.246608 26646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a270bb4b27942abaf964ecfd39565ee" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39997 } }
I20260923 22:21:24.246789 26646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:24.246829 26646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a270bb4b27942abaf964ecfd39565ee, State: Initialized, Role: FOLLOWER
I20260923 22:21:24.246949 26646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a270bb4b27942abaf964ecfd39565ee" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39997 } }
I20260923 22:21:24.247035 26646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:24.247071 26646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:24.247117 26646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:24.247790 26646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a270bb4b27942abaf964ecfd39565ee" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39997 } }
I20260923 22:21:24.247900 26646 leader_election.cc:302] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a270bb4b27942abaf964ecfd39565ee; no voters:
I20260923 22:21:24.248066 26646 leader_election.cc:288] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:24.248100 26651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:24.248083 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:24.242493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36685 (local address 127.4.207.62:39997)
0923 22:21:24.242673 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:24.242683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:24.242716 (+ 33us) server_negotiation.cc:407] Connection header received
0923 22:21:24.243157 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:24.243166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:24.243225 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:24.243352 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:24.243617 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:24.245008 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:24.246154 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:24.246525 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:24.246600 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:24.247400 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:24.247403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:24.247403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:24.247634 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:24.247662 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:24.247673 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:24.247738 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:24.247873 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 22:21:24.247898 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260923 22:21:24.248220 26651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [term 1 LEADER]: Becoming Leader. State: Replica: 1a270bb4b27942abaf964ecfd39565ee, State: Running, Role: LEADER
I20260923 22:21:24.248297 26646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:24.248287 26651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a270bb4b27942abaf964ecfd39565ee" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39997 } }
I20260923 22:21:24.248551 26651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a270bb4b27942abaf964ecfd39565ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a270bb4b27942abaf964ecfd39565ee" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39997 } } }
I20260923 22:21:24.248612 26651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:24.248600 26652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a270bb4b27942abaf964ecfd39565ee. Latest consensus state: current_term: 1 leader_uuid: "1a270bb4b27942abaf964ecfd39565ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a270bb4b27942abaf964ecfd39565ee" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 39997 } } }
I20260923 22:21:24.248690 26652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:24.248838 26657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:24.249045 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:24.249364 26657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:24.251040 26657 catalog_manager.cc:1347] Generated new cluster ID: 12739bf124154faaaca9eb3990bbffef
I20260923 22:21:24.251086 26657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:24.254014 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:24.250517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48363 (local address 127.4.207.62:39997)
0923 22:21:24.250577 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:24.250579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:24.250835 (+ 256us) server_negotiation.cc:407] Connection header received
0923 22:21:24.250899 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:24.250899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:24.250909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:24.250969 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:24.251148 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:24.251855 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:24.252803 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:24.253074 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:24.253143 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:24.253777 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:24.253778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:24.253778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:24.253782 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:24.253795 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:24.253799 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:24.253850 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:24.253941 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 22:21:24.253950 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 22:21:24.259027 26657 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:24.259634 26657 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:24.262825 26657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a270bb4b27942abaf964ecfd39565ee: Generated new TSK 0
I20260923 22:21:24.262987 26657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:24.266448 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26578
W20260923 22:21:24.270485 26563 connection.cc:535] client connection to 127.4.207.62:39997 recv error: Network error: failed to read from TLS socket (remote: 127.4.207.62:39997): Connection reset by peer (error 104)
2026-09-23T22:21:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1266 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:24 dist-test-slave-7511 krb5kdc[26677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:24 dist-test-slave-7511 krb5kdc[26677](info): set up 2 sockets
Sep 23 22:21:24 dist-test-slave-7511 krb5kdc[26677](info): commencing operation
krb5kdc: starting...
W20260923 22:21:25.279508 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:25 dist-test-slave-7511 krb5kdc[26677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:25.396049 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:40661
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:40661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:25.474200 26690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:25.474416 26690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:25.474452 26690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:25.475509 26690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:25.475574 26690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:25.475594 26690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:25.475613 26690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:25.475648 26690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:25.477123 26690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:40661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:40661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:25.477396 26690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:25.477608 26690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:25.482437 26698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:25.482556 26696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:25.482437 26695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:25.482714 26690 server_base.cc:1034] running on GCE node
I20260923 22:21:25.482954 26690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:25.483263 26690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:25.484424 26690 hybrid_clock.cc:648] HybridClock initialized: now 1790202085484418 us; error 30 us; skew 500 ppm
Sep 23 22:21:25 dist-test-slave-7511 krb5kdc[26677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202085, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:25.486678 26690 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:25.487074 26690 webserver.cc:460] Webserver started at http://127.4.207.62:43449/ using document root <none> and password file <none>
I20260923 22:21:25.487323 26690 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:25.487396 26690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:25.487556 26690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:25.488533 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "3cf67a83050d495a965dd33ab0a2a1da"
format_stamp: "Formatted at 2026-09-23 22:21:25 on dist-test-slave-7511"
server_key: "b8c2a576d6a8ed14e94abb7245d768e1"
server_key_iv: "bf86f16577b29b599051d6d466fa57a4"
server_key_version: "encryptionkey@0"
I20260923 22:21:25.488879 26690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "3cf67a83050d495a965dd33ab0a2a1da"
format_stamp: "Formatted at 2026-09-23 22:21:25 on dist-test-slave-7511"
server_key: "b8c2a576d6a8ed14e94abb7245d768e1"
server_key_iv: "bf86f16577b29b599051d6d466fa57a4"
server_key_version: "encryptionkey@0"
I20260923 22:21:25.490217 26690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:25.490968 26705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:25.491154 26690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:25.491273 26690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "3cf67a83050d495a965dd33ab0a2a1da"
format_stamp: "Formatted at 2026-09-23 22:21:25 on dist-test-slave-7511"
server_key: "b8c2a576d6a8ed14e94abb7245d768e1"
server_key_iv: "bf86f16577b29b599051d6d466fa57a4"
server_key_version: "encryptionkey@0"
I20260923 22:21:25.491364 26690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:25.494390 26690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:25.496203 26690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:25.496343 26690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:25.500082 26690 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:40661
I20260923 22:21:25.500168 26757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:40661 every 8 connection(s)
I20260923 22:21:25.500483 26690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:25.501427 26758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:25.503623 26758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da: Bootstrap starting.
I20260923 22:21:25.504136 26758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:25.504354 26758 log.cc:824] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:25.504985 26758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da: No bootstrap required, opened a new log
I20260923 22:21:25.506471 26758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf67a83050d495a965dd33ab0a2a1da" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40661 } }
I20260923 22:21:25.506598 26758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:25.506619 26758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf67a83050d495a965dd33ab0a2a1da, State: Initialized, Role: FOLLOWER
I20260923 22:21:25.506747 26758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf67a83050d495a965dd33ab0a2a1da" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40661 } }
I20260923 22:21:25.506830 26758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:25.506852 26758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:25.506879 26758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:25.507690 26758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf67a83050d495a965dd33ab0a2a1da" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40661 } }
I20260923 22:21:25.507772 26758 leader_election.cc:302] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cf67a83050d495a965dd33ab0a2a1da; no voters:
I20260923 22:21:25.507884 26758 leader_election.cc:288] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:25.507957 26761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:25.508134 26761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [term 1 LEADER]: Becoming Leader. State: Replica: 3cf67a83050d495a965dd33ab0a2a1da, State: Running, Role: LEADER
I20260923 22:21:25.508183 26758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:25.508219 26761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf67a83050d495a965dd33ab0a2a1da" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40661 } }
I20260923 22:21:25.508601 26763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cf67a83050d495a965dd33ab0a2a1da. Latest consensus state: current_term: 1 leader_uuid: "3cf67a83050d495a965dd33ab0a2a1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf67a83050d495a965dd33ab0a2a1da" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40661 } } }
I20260923 22:21:25.508692 26763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:25.508590 26762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cf67a83050d495a965dd33ab0a2a1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf67a83050d495a965dd33ab0a2a1da" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 40661 } } }
I20260923 22:21:25.508735 26762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:25.508888 26768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:25.509435 26768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:25.510259 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26690
I20260923 22:21:25.510381 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:25.510602 4924 external_mini_cluster.cc:1428] Setting key 92e88f5cfc82c73ec36091586ffd42cb
I20260923 22:21:25.511113 26768 catalog_manager.cc:1347] Generated new cluster ID: c6b8d491aa0b4c2a9b0ab93bff651813
I20260923 22:21:25.511157 26768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:25.513027 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:25.511172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38347 (local address 127.4.207.62:40661)
0923 22:21:25.511657 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:25.511664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:25.511687 (+ 23us) server_negotiation.cc:407] Connection header received
0923 22:21:25.511771 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:25.511776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:25.511798 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:25.511933 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:25.512607 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:25.512611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:25.512611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:25.512616 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:25.512668 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:25.512679 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:25.512744 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:25.512814 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 22:21:25.512857 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":92,"threads_started":1}
I20260923 22:21:25.513837 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:25.520635 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:25.515728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45063 (local address 127.4.207.62:40661)
0923 22:21:25.515764 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:25.515766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:25.519619 (+ 3853us) server_negotiation.cc:407] Connection header received
0923 22:21:25.519700 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:25.519701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:25.519707 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:25.519761 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:25.520402 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:25.520403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:25.520403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:25.520408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:25.520422 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:25.520427 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:25.520484 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:25.520540 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 22:21:25.520555 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:21:25.524835 26768 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:25.524979 26768 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:25.535185 26768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cf67a83050d495a965dd33ab0a2a1da: Generated new TSK 0
I20260923 22:21:25.535380 26768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:25.545392 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26690
2026-09-23T22:21:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1278 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:25 dist-test-slave-7511 krb5kdc[26789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:25 dist-test-slave-7511 krb5kdc[26789](info): set up 2 sockets
Sep 23 22:21:25 dist-test-slave-7511 krb5kdc[26789](info): commencing operation
krb5kdc: starting...
W20260923 22:21:26.560412 4924 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:26 dist-test-slave-7511 krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:26.684393 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:43683
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:43683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:26.763063 26802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:26.763262 26802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:26.763284 26802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:26.764418 26802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:26.764456 26802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:26.764470 26802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:26.764482 26802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:26.764494 26802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:26.765883 26802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:43683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:43683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:26.766062 26802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:26.766216 26802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 22:21:26.770833 26802 server_base.cc:1034] running on GCE node
W20260923 22:21:26.770778 26808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:26.770809 26810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:26.770783 26807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:26.771186 26802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:26.771535 26802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:26.772672 26802 hybrid_clock.cc:648] HybridClock initialized: now 1790202086772662 us; error 25 us; skew 500 ppm
Sep 23 22:21:26 dist-test-slave-7511 krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202086, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:26.774986 26802 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:26.775475 26802 webserver.cc:460] Webserver started at http://127.4.207.62:43725/ using document root <none> and password file <none>
I20260923 22:21:26.775720 26802 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:26.775785 26802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:26.775921 26802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:26.776912 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "f5df0752e1634e6cb4579b0f3d4feae3"
format_stamp: "Formatted at 2026-09-23 22:21:26 on dist-test-slave-7511"
server_key: "d0149c61cc22abc40a760a31a6e96a28"
server_key_iv: "c65d519e5df2e586ee60ce2c1343ad60"
server_key_version: "encryptionkey@0"
I20260923 22:21:26.777264 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "f5df0752e1634e6cb4579b0f3d4feae3"
format_stamp: "Formatted at 2026-09-23 22:21:26 on dist-test-slave-7511"
server_key: "d0149c61cc22abc40a760a31a6e96a28"
server_key_iv: "c65d519e5df2e586ee60ce2c1343ad60"
server_key_version: "encryptionkey@0"
I20260923 22:21:26.778546 26802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:26.779218 26817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:26.779402 26802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:26.779518 26802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "f5df0752e1634e6cb4579b0f3d4feae3"
format_stamp: "Formatted at 2026-09-23 22:21:26 on dist-test-slave-7511"
server_key: "d0149c61cc22abc40a760a31a6e96a28"
server_key_iv: "c65d519e5df2e586ee60ce2c1343ad60"
server_key_version: "encryptionkey@0"
I20260923 22:21:26.779599 26802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:26.806015 26802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:26.808167 26802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:26.808332 26802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:26.812433 26802 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:43683
I20260923 22:21:26.812469 26869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:43683 every 8 connection(s)
I20260923 22:21:26.812840 26802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:26.813684 26870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:26.815985 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3: Bootstrap starting.
I20260923 22:21:26.816524 26870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:26.816790 26870 log.cc:824] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:26.817461 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3: No bootstrap required, opened a new log
I20260923 22:21:26.819067 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26802
I20260923 22:21:26.819176 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:26.819190 26870 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df0752e1634e6cb4579b0f3d4feae3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43683 } }
I20260923 22:21:26.819366 4924 external_mini_cluster.cc:1428] Setting key fa3eb64be60881ee205c201b8cc34002
I20260923 22:21:26.819365 26870 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:26.819402 26870 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5df0752e1634e6cb4579b0f3d4feae3, State: Initialized, Role: FOLLOWER
I20260923 22:21:26.819577 26870 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df0752e1634e6cb4579b0f3d4feae3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43683 } }
I20260923 22:21:26.819646 26870 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:26.819700 26870 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:26.819751 26870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:26.820469 26870 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df0752e1634e6cb4579b0f3d4feae3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43683 } }
I20260923 22:21:26.820592 26870 leader_election.cc:302] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5df0752e1634e6cb4579b0f3d4feae3; no voters:
I20260923 22:21:26.820782 26870 leader_election.cc:288] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:26.820812 26875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:26.820982 26875 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [term 1 LEADER]: Becoming Leader. State: Replica: f5df0752e1634e6cb4579b0f3d4feae3, State: Running, Role: LEADER
I20260923 22:21:26.821062 26870 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:26.821081 26875 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df0752e1634e6cb4579b0f3d4feae3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43683 } }
I20260923 22:21:26.821404 26875 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5df0752e1634e6cb4579b0f3d4feae3. Latest consensus state: current_term: 1 leader_uuid: "f5df0752e1634e6cb4579b0f3d4feae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df0752e1634e6cb4579b0f3d4feae3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43683 } } }
I20260923 22:21:26.821498 26875 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:26.821413 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5df0752e1634e6cb4579b0f3d4feae3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5df0752e1634e6cb4579b0f3d4feae3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 43683 } } }
I20260923 22:21:26.821705 26881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:26.822278 26881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:26.821691 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:26.824036 26881 catalog_manager.cc:1347] Generated new cluster ID: 47f96ea1f9614df0854387b0e05ea4ad
I20260923 22:21:26.824092 26881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 22:21:26 dist-test-slave-7511 krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:26.830065 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:26.819877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43175 (local address 127.4.207.62:43683)
0923 22:21:26.820067 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:26.820076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:26.820110 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:26.820448 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:26.820454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:26.820498 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:26.820646 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:26.821475 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:26.822924 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:26.823839 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:26.824392 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:26.824523 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:26.826758 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:26.826761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:26.826762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:26.826773 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:26.828474 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:26.828863 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:26.828866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:26.828867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:26.828982 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:26.829192 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:26.829193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:26.829194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:26.829422 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:26.829618 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:26.829720 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 22:21:26.829870 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260923 22:21:26.831040 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:26.837318 26881 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:26.837971 26881 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:26.841737 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:26.832401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40693 (local address 127.4.207.62:43683)
0923 22:21:26.832439 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:26.832440 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:26.835635 (+ 3195us) server_negotiation.cc:407] Connection header received
0923 22:21:26.835716 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:26.835717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:26.835728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:26.835784 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:26.836288 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:26.836765 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:26.837683 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:26.837895 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:26.837979 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:26.839159 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:26.839160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:26.839160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:26.839167 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:26.840595 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:26.840961 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:26.840963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:26.840963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:26.841017 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:26.841223 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:26.841224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:26.841224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:26.841332 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:26.841451 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:26.841552 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 22:21:26.841634 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:26.850862 26881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5df0752e1634e6cb4579b0f3d4feae3: Generated new TSK 0
I20260923 22:21:26.851037 26881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:26.852783 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26802
2026-09-23T22:21:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:26 dist-test-slave-7511 krb5kdc[26901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:26 dist-test-slave-7511 krb5kdc[26901](info): set up 2 sockets
Sep 23 22:21:26 dist-test-slave-7511 krb5kdc[26901](info): commencing operation
krb5kdc: starting...
W20260923 22:21:27.879621 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:27 dist-test-slave-7511 krb5kdc[26901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:27.992535 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35515
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:28.074184 26914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:28.074384 26914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:28.074406 26914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:28.075619 26914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:28.075662 26914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:28.075676 26914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:28.075690 26914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:28.075702 26914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:28.077265 26914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:28.077521 26914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:28.077757 26914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:28.082628 26922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:28.082657 26919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:28.082631 26920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:28.082846 26914 server_base.cc:1034] running on GCE node
I20260923 22:21:28.083168 26914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:28.083421 26914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:28.084666 26914 hybrid_clock.cc:648] HybridClock initialized: now 1790202088084650 us; error 30 us; skew 500 ppm
Sep 23 22:21:28 dist-test-slave-7511 krb5kdc[26901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202088, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:28.092095 26914 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:28.092528 26914 webserver.cc:460] Webserver started at http://127.4.207.62:37177/ using document root <none> and password file <none>
I20260923 22:21:28.092754 26914 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:28.092823 26914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:28.092967 26914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:28.094000 26914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "d2bde130f3fd4e7f96206024a6ce42cf"
format_stamp: "Formatted at 2026-09-23 22:21:28 on dist-test-slave-7511"
server_key: "d3a9451916d82f0b2566564bb8d48bf4"
server_key_iv: "65def15712c070ba9bb140ddddf5654f"
server_key_version: "encryptionkey@0"
I20260923 22:21:28.094363 26914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "d2bde130f3fd4e7f96206024a6ce42cf"
format_stamp: "Formatted at 2026-09-23 22:21:28 on dist-test-slave-7511"
server_key: "d3a9451916d82f0b2566564bb8d48bf4"
server_key_iv: "65def15712c070ba9bb140ddddf5654f"
server_key_version: "encryptionkey@0"
I20260923 22:21:28.095737 26914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:28.096489 26929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:28.096743 26914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 22:21:28.096839 26914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "d2bde130f3fd4e7f96206024a6ce42cf"
format_stamp: "Formatted at 2026-09-23 22:21:28 on dist-test-slave-7511"
server_key: "d3a9451916d82f0b2566564bb8d48bf4"
server_key_iv: "65def15712c070ba9bb140ddddf5654f"
server_key_version: "encryptionkey@0"
I20260923 22:21:28.096957 26914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:28.110495 26914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:28.112526 26914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:28.112694 26914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:28.116838 26914 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:35515
I20260923 22:21:28.116915 26981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:35515 every 8 connection(s)
I20260923 22:21:28.117298 26914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:28.118178 26982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:28.120389 26982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf: Bootstrap starting.
I20260923 22:21:28.120965 26982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:28.121239 26982 log.cc:824] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:28.121912 26982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf: No bootstrap required, opened a new log
I20260923 22:21:28.123418 26982 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bde130f3fd4e7f96206024a6ce42cf" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35515 } }
I20260923 22:21:28.123572 26982 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:28.123618 26982 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bde130f3fd4e7f96206024a6ce42cf, State: Initialized, Role: FOLLOWER
I20260923 22:21:28.123813 26982 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bde130f3fd4e7f96206024a6ce42cf" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35515 } }
I20260923 22:21:28.123888 26982 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:28.123932 26982 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:28.123976 26982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:28.124728 26982 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bde130f3fd4e7f96206024a6ce42cf" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35515 } }
I20260923 22:21:28.124833 26982 leader_election.cc:302] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2bde130f3fd4e7f96206024a6ce42cf; no voters:
I20260923 22:21:28.125005 26982 leader_election.cc:288] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:28.125029 26985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:28.125181 26985 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [term 1 LEADER]: Becoming Leader. State: Replica: d2bde130f3fd4e7f96206024a6ce42cf, State: Running, Role: LEADER
I20260923 22:21:28.125231 26982 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:28.125298 26985 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bde130f3fd4e7f96206024a6ce42cf" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35515 } }
I20260923 22:21:28.125614 26986 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2bde130f3fd4e7f96206024a6ce42cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bde130f3fd4e7f96206024a6ce42cf" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35515 } } }
I20260923 22:21:28.125622 26987 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2bde130f3fd4e7f96206024a6ce42cf. Latest consensus state: current_term: 1 leader_uuid: "d2bde130f3fd4e7f96206024a6ce42cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2bde130f3fd4e7f96206024a6ce42cf" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35515 } } }
I20260923 22:21:28.125699 26986 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:28.125706 26987 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:28.125902 26993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:28.126546 26993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:28.127118 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 26914
I20260923 22:21:28.127235 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:28.127429 4924 external_mini_cluster.cc:1428] Setting key f9836f333cf205210f4c7c6192fea1de
I20260923 22:21:28.132726 26993 catalog_manager.cc:1347] Generated new cluster ID: b02403502a114efe9cdc3bb5be1efc56
I20260923 22:21:28.132813 26993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:28.133230 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:28.127906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51003 (local address 127.4.207.62:35515)
0923 22:21:28.128076 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:28.128084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:28.128113 (+ 29us) server_negotiation.cc:407] Connection header received
0923 22:21:28.128239 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:28.128246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:28.128309 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:28.128470 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:28.128704 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.130102 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.130892 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.131575 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.131718 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:28.132409 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:28.132413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:28.132414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:28.132609 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:28.132646 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:28.132660 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:28.132769 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:28.132969 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 22:21:28.133032 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260923 22:21:28.134356 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:28.139132 27004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:28.135963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55163 (local address 127.4.207.62:35515)
0923 22:21:28.135992 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:28.135993 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:28.136001 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:28.136060 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:28.136061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:28.136070 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:28.136121 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:28.136258 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.136943 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.137950 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:28.138227 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:28.138281 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:28.138910 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:28.138911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:28.138911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:28.138916 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:28.138928 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:28.138932 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:28.138980 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:28.139045 (+ 65us) server_negotiation.cc:299] Negotiation successful
0923 22:21:28.139065 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260923 22:21:28.144798 26993 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:28.145485 26993 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:28.152083 26993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2bde130f3fd4e7f96206024a6ce42cf: Generated new TSK 0
I20260923 22:21:28.152273 26993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:28.158030 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 26914
2026-09-23T22:21:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:28 dist-test-slave-7511 krb5kdc[27013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:28 dist-test-slave-7511 krb5kdc[27013](info): set up 2 sockets
Sep 23 22:21:28 dist-test-slave-7511 krb5kdc[27013](info): commencing operation
krb5kdc: starting...
W20260923 22:21:29.179421 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:29 dist-test-slave-7511 krb5kdc[27013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:29.294534 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:44681
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:44681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:29.373788 27026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:29.373997 27026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:29.374022 27026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:29.375141 27026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:29.375187 27026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:29.375202 27026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:29.375216 27026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:29.375228 27026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:29.376680 27026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:44681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:44681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:29.376876 27026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:29.377048 27026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:29.381913 27034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:29.381927 27031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:29.381973 27026 server_base.cc:1034] running on GCE node
W20260923 22:21:29.381918 27032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:29.382418 27026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:29.382705 27026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:29.383842 27026 hybrid_clock.cc:648] HybridClock initialized: now 1790202089383826 us; error 42 us; skew 500 ppm
Sep 23 22:21:29 dist-test-slave-7511 krb5kdc[27013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202089, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:29.386199 27026 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:29.386596 27026 webserver.cc:460] Webserver started at http://127.4.207.62:36759/ using document root <none> and password file <none>
I20260923 22:21:29.386803 27026 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:29.386868 27026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:29.387006 27026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:29.388041 27026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "31b380d167544ea794094d583760e5d3"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-7511"
server_key: "e60f733a74663bf1e4b9eae5943a11a1"
server_key_iv: "8a35d61b9720dd2977a6614925c1e5a9"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.388420 27026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "31b380d167544ea794094d583760e5d3"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-7511"
server_key: "e60f733a74663bf1e4b9eae5943a11a1"
server_key_iv: "8a35d61b9720dd2977a6614925c1e5a9"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.389694 27026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:29.390422 27041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:29.390614 27026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:29.390717 27026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "31b380d167544ea794094d583760e5d3"
format_stamp: "Formatted at 2026-09-23 22:21:29 on dist-test-slave-7511"
server_key: "e60f733a74663bf1e4b9eae5943a11a1"
server_key_iv: "8a35d61b9720dd2977a6614925c1e5a9"
server_key_version: "encryptionkey@0"
I20260923 22:21:29.390798 27026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:29.393908 27026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:29.395881 27026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:29.396291 27026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:29.400137 27026 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:44681
I20260923 22:21:29.400171 27093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:44681 every 8 connection(s)
I20260923 22:21:29.400502 27026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:29.401450 27094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:29.403607 27094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3: Bootstrap starting.
I20260923 22:21:29.404156 27094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:29.404434 27094 log.cc:824] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:29.405107 27094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3: No bootstrap required, opened a new log
I20260923 22:21:29.406663 27094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b380d167544ea794094d583760e5d3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44681 } }
I20260923 22:21:29.406790 27094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 22:21:29.406812 27094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b380d167544ea794094d583760e5d3, State: Initialized, Role: FOLLOWER
I20260923 22:21:29.406980 27094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b380d167544ea794094d583760e5d3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44681 } }
I20260923 22:21:29.407058 27094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 22:21:29.407099 27094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 22:21:29.407150 27094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:29.408002 27094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b380d167544ea794094d583760e5d3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44681 } }
I20260923 22:21:29.408108 27094 leader_election.cc:302] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31b380d167544ea794094d583760e5d3; no voters:
I20260923 22:21:29.408284 27094 leader_election.cc:288] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 22:21:29.408350 27097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:29.408557 27097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [term 1 LEADER]: Becoming Leader. State: Replica: 31b380d167544ea794094d583760e5d3, State: Running, Role: LEADER
I20260923 22:21:29.408581 27094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:29.408771 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 27026
I20260923 22:21:29.408937 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:29.409102 4924 external_mini_cluster.cc:1428] Setting key cc2559105e4c11dbce93c0cfbe103b8b
I20260923 22:21:29.408661 27097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b380d167544ea794094d583760e5d3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44681 } }
I20260923 22:21:29.409782 27099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31b380d167544ea794094d583760e5d3. Latest consensus state: current_term: 1 leader_uuid: "31b380d167544ea794094d583760e5d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b380d167544ea794094d583760e5d3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44681 } } }
I20260923 22:21:29.409884 27099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:29.410023 27098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31b380d167544ea794094d583760e5d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b380d167544ea794094d583760e5d3" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 44681 } } }
I20260923 22:21:29.410181 27098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3 [sys.catalog]: This master's current role is: LEADER
W20260923 22:21:29.411186 27114 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:29.411384 27114 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:29.411494 27108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:29.411729 27108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:29.412034 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:29.409747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49807 (local address 127.4.207.62:44681)
0923 22:21:29.409943 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:29.409954 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:29.409988 (+ 34us) server_negotiation.cc:407] Connection header received
0923 22:21:29.410246 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:29.410253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:29.410284 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:29.410404 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:29.411591 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:29.411594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:29.411594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:29.411599 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:29.411636 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:29.411663 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:29.411721 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:29.411837 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 22:21:29.411874 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260923 22:21:29.413017 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:29.413421 27108 catalog_manager.cc:1347] Generated new cluster ID: 0c6f4fbb80c24fd189b182ab75cde41f
I20260923 22:21:29.413478 27108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:29.420893 27105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:29.419716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59807 (local address 127.4.207.62:44681)
0923 22:21:29.419749 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:29.419751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:29.419915 (+ 164us) server_negotiation.cc:407] Connection header received
0923 22:21:29.419949 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:29.419950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:29.419956 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:29.420020 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:29.420659 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:29.420660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:29.420661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 22:21:29.420665 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:29.420681 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 22:21:29.420685 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:29.420745 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:29.420807 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 22:21:29.420820 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 22:21:29.427284 27108 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:29.427461 27108 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:29.433940 27108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31b380d167544ea794094d583760e5d3: Generated new TSK 0
I20260923 22:21:29.434139 27108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:29.436506 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 27026
2026-09-23T22:21:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1278 ms)
[----------] 18 tests from AuthTokenIssuingTest (23321 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:29 dist-test-slave-7511 krb5kdc[27125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:29 dist-test-slave-7511 krb5kdc[27125](info): set up 2 sockets
Sep 23 22:21:29 dist-test-slave-7511 krb5kdc[27125](info): commencing operation
krb5kdc: starting...
W20260923 22:21:30.455952 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:30 dist-test-slave-7511 krb5kdc[27125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:30.574316 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35991
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35991,127.4.207.61:40465,127.4.207.60:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:30.654076 27138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:30.654289 27138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:30.654305 27138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:30.654325 27138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:30.655493 27138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:30.655534 27138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:30.655546 27138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:30.655558 27138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:30.655594 27138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:30.657007 27138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35991,127.4.207.61:40465,127.4.207.60:42007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:30.657253 27138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:30.657486 27138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:30.662209 27146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:30.662391 27138 server_base.cc:1034] running on GCE node
W20260923 22:21:30.662241 27144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:30.662230 27143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:30.662689 27138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:30.662984 27138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:30.664115 27138 hybrid_clock.cc:648] HybridClock initialized: now 1790202090664096 us; error 36 us; skew 500 ppm
Sep 23 22:21:30 dist-test-slave-7511 krb5kdc[27125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:30.666551 27138 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:30.667012 27138 webserver.cc:460] Webserver started at http://127.4.207.62:37717/ using document root <none> and password file <none>
I20260923 22:21:30.667230 27138 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:30.667270 27138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:30.667425 27138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:30.668484 27138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "3f432b564e0f42a39dac0d8f2b5055ec"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "55b0bad7f26c3675fd830696ef715e1c"
server_key_iv: "3528a3d1714703b5da8e66e45b1290f5"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.668829 27138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "3f432b564e0f42a39dac0d8f2b5055ec"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "55b0bad7f26c3675fd830696ef715e1c"
server_key_iv: "3528a3d1714703b5da8e66e45b1290f5"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.670164 27138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:21:30.670925 27153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:30.671134 27138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:30.671233 27138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "3f432b564e0f42a39dac0d8f2b5055ec"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "55b0bad7f26c3675fd830696ef715e1c"
server_key_iv: "3528a3d1714703b5da8e66e45b1290f5"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.671314 27138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:30.684127 27138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:30.685801 27138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:30.685961 27138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:30.690132 27138 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:35991
I20260923 22:21:30.690196 27205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:35991 every 8 connection(s)
I20260923 22:21:30.690533 27138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:30.691573 27206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:30.693140 27206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:30 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:30.698925 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 27138
I20260923 22:21:30.699057 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:30.699254 4924 external_mini_cluster.cc:1428] Setting key 7f9a90fdd8461c5fd7a92cbcc55b7436
I20260923 22:21:30.702637 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.693729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35991 (local address 127.0.0.1:50540)
0923 22:21:30.693941 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.693973 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.694438 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.694689 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.694692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.694932 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.695088 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.695092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.696137 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.696137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.696798 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.696812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.697265 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.697266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.697321 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.697773 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.697780 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.699294 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.701168 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.701174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.701176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.701365 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.701619 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.701620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.701621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.701714 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.702053 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.702058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.702117 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.702231 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.702470 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":97,"thread_start_us":128,"threads_started":1}
I20260923 22:21:30.702641 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.693802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.4.207.62:35991)
0923 22:21:30.693966 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.693975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.694060 (+ 85us) server_negotiation.cc:407] Connection header received
0923 22:21:30.694566 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.694568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.694584 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.694657 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.695196 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.696033 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.696880 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.697169 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.697240 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.699385 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.699392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.699393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.699399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.701024 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.701460 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.701463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.701463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.701547 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.701780 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.701780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.701781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.701980 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.702073 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.702264 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.702397 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260923 22:21:30.703610 27206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:30.703990 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.703858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42164)
0923 22:21:30.703906 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.703948 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 22:21:30.704102 27155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
I20260923 22:21:30.704548 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.704462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42176)
0923 22:21:30.704506 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.704529 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:21:30.704654 27206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:40465: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.207.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab.
Entry for principal kudu/127.4.207.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab.
I20260923 22:21:30.722203 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.61:40465
--webserver_interface=127.4.207.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab
--principal=kudu/127.4.207.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35991,127.4.207.61:40465,127.4.207.60:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:21:30.753866 27206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } attempt: 1
I20260923 22:21:30.754383 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.754214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42184)
0923 22:21:30.754289 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.754341 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 22:21:30.754762 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.754670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42194)
0923 22:21:30.754716 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.754744 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:21:30.754884 27206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:40465: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
W20260923 22:21:30.801945 27213 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:30.802160 27213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:30.802176 27213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:30.802196 27213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:30.803328 27213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:30.803377 27213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:30.803392 27213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:30.803404 27213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:30.803417 27213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:30.804893 27213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35991,127.4.207.61:40465,127.4.207.60:42007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab
--principal=kudu/127.4.207.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.61:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.207.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:30.805133 27213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:30.805379 27213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:30.810191 27218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:30.810205 27221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:30.810305 27219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:30.810353 27213 server_base.cc:1034] running on GCE node
I20260923 22:21:30.810688 27213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:30.811019 27213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:30.812162 27213 hybrid_clock.cc:648] HybridClock initialized: now 1790202090812139 us; error 39 us; skew 500 ppm
Sep 23 22:21:30 dist-test-slave-7511 krb5kdc[27125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:30.814642 27213 init.cc:377] Logged in from keytab as kudu/127.4.207.61@KRBTEST.COM (short username kudu)
I20260923 22:21:30.815056 27213 webserver.cc:460] Webserver started at http://127.4.207.61:38431/ using document root <none> and password file <none>
I20260923 22:21:30.815297 27213 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:30.815343 27213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:30.815503 27213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:30.816522 27213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/data/instance:
uuid: "f67c69b979524a6aa0a7a2f214151976"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "90cd1443229c85b8187c68f284a72f17"
server_key_iv: "216ace88b96a625e488bf138dc82f48c"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.816890 27213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/wal/instance:
uuid: "f67c69b979524a6aa0a7a2f214151976"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "90cd1443229c85b8187c68f284a72f17"
server_key_iv: "216ace88b96a625e488bf138dc82f48c"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.818217 27213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 22:21:30.819077 27228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:30.819265 27213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 22:21:30.819396 27213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/wal
uuid: "f67c69b979524a6aa0a7a2f214151976"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "90cd1443229c85b8187c68f284a72f17"
server_key_iv: "216ace88b96a625e488bf138dc82f48c"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.819518 27213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:30.823081 27206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } attempt: 2
I20260923 22:21:30.823570 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.823403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42206)
0923 22:21:30.823507 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.823542 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 22:21:30.823895 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.823827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42216)
0923 22:21:30.823860 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.823881 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260923 22:21:30.824000 27206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:40465: Network error: Client connection negotiation failed: client connection to 127.4.207.61:40465: connect: Connection refused (error 111)
I20260923 22:21:30.840597 27213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:30.842512 27213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:30.842675 27213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:30.846585 27213 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.61:40465
I20260923 22:21:30.846699 27280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.61:40465 every 8 connection(s)
I20260923 22:21:30.847010 27213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
I20260923 22:21:30.848237 27281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:30.850011 27281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:30 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:30.856818 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 27213
I20260923 22:21:30.856986 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-1/wal/instance
I20260923 22:21:30.857259 4924 external_mini_cluster.cc:1428] Setting key bae73e6908b6af92325642d8ae8d053d
I20260923 22:21:30.859982 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.850774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35991 (local address 127.0.0.1:50546)
0923 22:21:30.850957 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.851002 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.851602 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.851840 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.851843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.852108 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.852325 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.852330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.852984 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.852985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.853925 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.853928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.854293 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.854294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.854404 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.854987 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.854995 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.856579 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.858304 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.858312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.858314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.858683 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.858972 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.858972 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.858973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.859073 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.859375 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.859380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.859563 (+ 183us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.859678 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.859809 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
I20260923 22:21:30.860006 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.850824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.4.207.62:35991)
0923 22:21:30.850872 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.850874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.851099 (+ 225us) server_negotiation.cc:407] Connection header received
0923 22:21:30.851711 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.851712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.851724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.851782 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.852414 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.852893 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.854003 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.854196 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.854253 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.856675 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.856677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.856677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.856682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.858214 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.858810 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.858812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.858813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.858900 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.859146 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.859147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.859148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.859298 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.859372 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.859725 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.859813 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:21:30.860670 27281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.207.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.60@KRBTEST.COM" created.
Sep 23 22:21:30 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:21:30.872301 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.860952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42222)
0923 22:21:30.861007 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.861019 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.861066 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.861452 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.861452 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.861571 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.861624 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.861626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.862944 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.862945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.863542 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.863544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.864195 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.864196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.864259 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.864714 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.864719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.869147 (+ 4428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.871095 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.871096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.871096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.871298 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.871595 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.871596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.871597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.871656 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.871953 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.871954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.871992 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.872060 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.872164 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 22:21:30.872368 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.861015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.4.207.61:40465)
0923 22:21:30.861270 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.861276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.861302 (+ 26us) server_negotiation.cc:407] Connection header received
0923 22:21:30.861326 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.861328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.861345 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.861438 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.861713 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.862854 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.863621 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.864109 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.864187 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.869263 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.869272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.869273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.869280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.871012 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.871410 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.871414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.871416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.871528 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.871735 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.871736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.871737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.871882 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.871974 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.872098 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.872223 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":222,"thread_start_us":125,"threads_started":1}
I20260923 22:21:30.872932 27281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:30.873291 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.873168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48068)
0923 22:21:30.873219 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.873264 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:21:30.873428 27229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
I20260923 22:21:30.873842 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.873753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48070)
0923 22:21:30.873796 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.873820 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:21:30.873957 27281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:42007: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab.
Entry for principal kudu/127.4.207.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab.
I20260923 22:21:30.884004 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.60:42007
--webserver_interface=127.4.207.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab
--principal=kudu/127.4.207.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:35991,127.4.207.61:40465,127.4.207.60:42007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:21:30.915192 27206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } attempt: 3
Sep 23 22:21:30 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:21:30.922640 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.915624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42230)
0923 22:21:30.915688 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.915700 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:30.915792 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:30.915947 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:30.915948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:30.916109 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:30.916176 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.916178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.916868 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.916869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.917405 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:30.917407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.917821 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.917822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:30.917903 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.918388 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:30.918398 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:30.919991 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:30.921607 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.921608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.921608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.921797 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.922027 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:30.922028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:30.922029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:30.922086 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:30.922332 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:30.922333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:30.922378 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 22:21:30.922450 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 22:21:30.922544 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 22:21:30.922640 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.915705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.4.207.61:40465)
0923 22:21:30.915797 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:30.915799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:30.915826 (+ 27us) server_negotiation.cc:407] Connection header received
0923 22:21:30.915854 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:30.915855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:30.915874 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:30.915928 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:30.916264 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.916760 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.917494 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:30.917737 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:30.917826 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:30.920077 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:30.920078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:30.920078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:30.920086 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:30.921502 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.921866 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.921867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.921868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.921925 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:30.922148 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:30.922149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:30.922149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:30.922265 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:30.922339 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:30.922472 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 22:21:30.922544 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 22:21:30.923074 27206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:30.923091 27281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } attempt: 1
I20260923 22:21:30.923478 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.923336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48102)
0923 22:21:30.923371 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.923404 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260923 22:21:30.923620 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.923317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48086)
0923 22:21:30.923559 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.923589 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227}
I20260923 22:21:30.923781 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.923698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48118)
0923 22:21:30.923733 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.923759 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260923 22:21:30.923971 27281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:42007: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
I20260923 22:21:30.924014 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.923918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48128)
0923 22:21:30.923964 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.923990 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 22:21:30.924075 27206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:42007: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
I20260923 22:21:30.955246 27206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } attempt: 1
I20260923 22:21:30.955747 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.955601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48140)
0923 22:21:30.955669 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.955714 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 22:21:30.956130 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.956032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48144)
0923 22:21:30.956078 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.956105 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:21:30.956218 27206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:42007: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
W20260923 22:21:30.965284 27288 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:30.965485 27288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:30.965503 27288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:30.965556 27288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:30.966668 27288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:30.966711 27288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:30.966725 27288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:30.966738 27288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:30.966750 27288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:30.968350 27288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:35991,127.4.207.61:40465,127.4.207.60:42007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab
--principal=kudu/127.4.207.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.60:42007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.207.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:30.968603 27288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:30.968837 27288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:30.973732 27293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:30.973809 27294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:30.973732 27296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:30.974216 27288 server_base.cc:1034] running on GCE node
I20260923 22:21:30.974437 27288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:30.974695 27288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:30.975832 27288 hybrid_clock.cc:648] HybridClock initialized: now 1790202090975821 us; error 27 us; skew 500 ppm
Sep 23 22:21:30 dist-test-slave-7511 krb5kdc[27125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:30.978415 27288 init.cc:377] Logged in from keytab as kudu/127.4.207.60@KRBTEST.COM (short username kudu)
I20260923 22:21:30.978870 27288 webserver.cc:460] Webserver started at http://127.4.207.60:36987/ using document root <none> and password file <none>
I20260923 22:21:30.979089 27288 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:30.979130 27288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:30.979239 27288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:30.980314 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/data/instance:
uuid: "c4fea4e331904944b8de83914964c771"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "da661aab1149308ef58bd56eb2ef8893"
server_key_iv: "810d2dd14c49f8ec4d746d266c23b698"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.980645 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/wal/instance:
uuid: "c4fea4e331904944b8de83914964c771"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "da661aab1149308ef58bd56eb2ef8893"
server_key_iv: "810d2dd14c49f8ec4d746d266c23b698"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.981897 27288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:21:30.982605 27303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:30.982828 27288 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 22:21:30.982925 27288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/wal
uuid: "c4fea4e331904944b8de83914964c771"
format_stamp: "Formatted at 2026-09-23 22:21:30 on dist-test-slave-7511"
server_key: "da661aab1149308ef58bd56eb2ef8893"
server_key_iv: "810d2dd14c49f8ec4d746d266c23b698"
server_key_version: "encryptionkey@0"
I20260923 22:21:30.983008 27288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:30.992115 27281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } attempt: 2
I20260923 22:21:30.992524 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.992401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48158)
0923 22:21:30.992459 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.992496 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 22:21:30.992858 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:30.992760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48168)
0923 22:21:30.992806 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:30.992833 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 22:21:30.992945 27281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:42007: Network error: Client connection negotiation failed: client connection to 127.4.207.60:42007: connect: Connection refused (error 111)
I20260923 22:21:30.994453 27288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:30.996503 27288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:30.996690 27288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:31.001093 27288 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.60:42007
I20260923 22:21:31.001125 27355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.60:42007 every 8 connection(s)
I20260923 22:21:31.001488 27288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
I20260923 22:21:31.002419 27356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:31.003993 27356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:31.008713 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 27288
I20260923 22:21:31.008809 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790201827590652-4924-0/minicluster-data/master-2/wal/instance
I20260923 22:21:31.009061 4924 external_mini_cluster.cc:1428] Setting key f04c30813b631aa4dfa1ff4498c5a2b9
Sep 23 22:21:31 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:31.014402 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.004697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.4.207.62:35991)
0923 22:21:31.004773 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.004775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.004977 (+ 202us) server_negotiation.cc:407] Connection header received
0923 22:21:31.005412 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.005413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.005424 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.005486 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.006116 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.006615 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.007763 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.007957 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.008023 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.010805 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.010807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.010807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.010815 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.012850 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.013371 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.013372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.013373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.013435 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.013735 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.013737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.013737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.013866 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.013944 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.014223 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.014310 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 22:21:31.014588 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.004666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:35991 (local address 127.0.0.1:50558)
0923 22:21:31.004857 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:31.004892 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:31.005325 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:31.005545 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:31.005548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:31.005824 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:31.006020 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.006024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.006727 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.006729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.007603 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.007605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.008043 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.008044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.008142 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.008669 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:31.008678 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:31.010688 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:31.012994 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.013002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.013004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.013271 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.013508 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.013509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.013509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.013635 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.013935 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:31.013942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:31.014085 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 22:21:31.014197 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 22:21:31.014343 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
Sep 23 22:21:31 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:21:31.015235 27356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:31.018399 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.009512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48172 (local address 127.4.207.60:42007)
0923 22:21:31.009650 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.009656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.009787 (+ 131us) server_negotiation.cc:407] Connection header received
0923 22:21:31.009839 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.009842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.009863 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.009946 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.010776 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.011778 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.012554 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.012898 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.012995 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.015511 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.015517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.015518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.015526 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.017245 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.017568 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.017571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.017572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.017638 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.017862 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.017862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.017862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.017979 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.018133 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.018241 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.018300 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
Sep 23 22:21:31 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:21:31.023065 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.015525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:40465 (local address 127.0.0.1:42232)
0923 22:21:31.015661 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:31.015671 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:31.015722 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:31.015897 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:31.015898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:31.016062 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:31.016100 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.016102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.016855 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.016856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.017576 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.017577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.018036 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.018037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.018094 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.018493 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:31.018498 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:31.019888 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:31.022080 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.022080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.022081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.022249 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.022459 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.022460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.022461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.022515 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.022761 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:31.022761 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:31.022801 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 22:21:31.022890 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 22:21:31.022975 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260923 22:21:31.023067 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.015579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.4.207.61:40465)
0923 22:21:31.015640 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.015641 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.015709 (+ 68us) server_negotiation.cc:407] Connection header received
0923 22:21:31.015782 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.015783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.015794 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.015866 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.016195 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.016775 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.017649 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.017965 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.018013 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.019971 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.019972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.019973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.019980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.021985 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.022335 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.022336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.022336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.022387 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.022586 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.022587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.022587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.022707 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.022768 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.022916 (+ 148us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.022982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:31.023610 27356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:31 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:31.027319 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.019509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.4.207.62:35991)
0923 22:21:31.019544 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.019546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.019552 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:21:31.019571 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.019572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.019580 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.019649 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.020158 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.020607 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.021830 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.022029 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.022096 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.024640 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.024641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.024641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.024647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.026134 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.026495 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.026495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.026496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.026567 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.026779 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.026780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.026781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.026917 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.027098 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.027139 (+ 41us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.027215 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 23 22:21:31 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:21:31.030848 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.023830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48182)
0923 22:21:31.023873 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:31.023882 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:31.023918 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:31.024032 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:31.024033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:31.024170 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:31.024199 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.024200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.025037 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.025038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.025864 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.025866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.026220 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.026221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.026271 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.026622 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:31.026627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:31.028060 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:31.029813 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.029814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.029814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.030009 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.030208 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.030209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.030209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.030271 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.030529 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:31.030529 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:31.030566 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 22:21:31.030641 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 22:21:31.030737 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 22:21:31.030903 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.023879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.4.207.60:42007)
0923 22:21:31.023911 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.023913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.023920 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:21:31.023972 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.023973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.023980 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.024029 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.024281 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.024970 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.025944 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.026135 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.026223 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.028129 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.028130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.028130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.028138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.029714 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.030085 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.030086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.030086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.030133 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.030331 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.030332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.030332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.030455 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.030523 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.030684 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.030787 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":18}
I20260923 22:21:31.031365 27206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } attempt: 2
I20260923 22:21:31.032361 27356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771: Bootstrap starting.
Sep 23 22:21:31 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:21:31.033360 27356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:31.033653 27356 log.cc:824] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:31.034492 27356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771: No bootstrap required, opened a new log
Sep 23 22:21:31 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:21:31.036733 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.027946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.4.207.61:40465)
0923 22:21:31.028002 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.028004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.028029 (+ 25us) server_negotiation.cc:407] Connection header received
0923 22:21:31.028050 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.028050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.028060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.028122 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.028691 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.029111 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.030013 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.030353 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.030417 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.033111 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.033113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.033113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.033120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.035334 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.035781 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.035782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.035783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.035834 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.036031 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.036032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.036033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.036166 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.036325 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.036519 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.036609 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 22:21:31.036703 27356 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } }
I20260923 22:21:31.036864 27356 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fea4e331904944b8de83914964c771, State: Initialized, Role: FOLLOWER
I20260923 22:21:31.036962 27356 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } }
I20260923 22:21:31.037382 27356 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:31.038067 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:31.038897 27208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.031633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48190)
0923 22:21:31.031682 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:31.031692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:31.031730 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:31.032062 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:31.032063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:31.032198 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:31.032229 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.032230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.032930 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.032931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.033749 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.033752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.034209 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.034210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.034271 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.034612 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:31.034617 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:31.035849 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:31.037687 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.037688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.037688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.037923 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.038193 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.038195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.038195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.038261 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.038582 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:31.038583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:31.038631 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 22:21:31.038701 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 22:21:31.038792 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:21:31.038875 27362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.039677 27362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:21:31.039366 27373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:31.039746 27373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:31.039499 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.031882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.4.207.60:42007)
0923 22:21:31.031922 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.031924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.031930 (+ 6us) server_negotiation.cc:407] Connection header received
0923 22:21:31.031949 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.031950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.031959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.032053 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.032302 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.032854 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.033826 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.034117 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.034202 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.035953 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.035953 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.035954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.035960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.037589 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.038038 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.038039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.038039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.038095 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.038336 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.038337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.038337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.038501 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.038571 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.039234 (+ 663us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.039332 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:31.039040 27180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3f432b564e0f42a39dac0d8f2b5055ec"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50572
I20260923 22:21:31.041250 27206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec: Bootstrap starting.
I20260923 22:21:31.042042 27206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:31.042351 27206 log.cc:824] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:31.043167 27206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec: No bootstrap required, opened a new log
I20260923 22:21:31.040236 27288 master_runner.cc:186] Error getting master registration for 127.4.207.62:35991: OK, instance_id { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" instance_seqno: 1790202090688897 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:21:31.043269 27288 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:21:31.044521 27206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } }
I20260923 22:21:31.044647 27206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f432b564e0f42a39dac0d8f2b5055ec, State: Initialized, Role: FOLLOWER
I20260923 22:21:31.044729 27206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } }
I20260923 22:21:31.044963 27376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.045018 27206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:31.045101 27376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:31.045040 27180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 22:21:31.046319 27180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:31.046969 27180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } }
I20260923 22:21:31.047501 27330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c4fea4e331904944b8de83914964c771"
I20260923 22:21:31.047636 27330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:31.048506 27330 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f432b564e0f42a39dac0d8f2b5055ec in term 1.
I20260923 22:21:31.048875 27180 leader_election.cc:288] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [CANDIDATE]: Term 1 election: Requested vote from peers f67c69b979524a6aa0a7a2f214151976 (127.4.207.61:40465), c4fea4e331904944b8de83914964c771 (127.4.207.60:42007)
I20260923 22:21:31.049224 27154 leader_election.cc:302] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f432b564e0f42a39dac0d8f2b5055ec, c4fea4e331904944b8de83914964c771; no voters:
I20260923 22:21:31.049347 27376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:31.049475 27376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [term 1 LEADER]: Becoming Leader. State: Replica: 3f432b564e0f42a39dac0d8f2b5055ec, State: Running, Role: LEADER
I20260923 22:21:31.049548 27376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } }
I20260923 22:21:31.049966 27376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f432b564e0f42a39dac0d8f2b5055ec. Latest consensus state: current_term: 1 leader_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.050064 27376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:31.048756 27138 master_runner.cc:186] Error getting master registration for 127.4.207.61:40465: OK, instance_id { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" instance_seqno: 1790202090845584 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:21:31.051047 27138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:21:31.051621 27394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:31.052139 27394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:31.060050 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.050726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.4.207.60:42007)
0923 22:21:31.050762 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.050763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.051810 (+ 1047us) server_negotiation.cc:407] Connection header received
0923 22:21:31.051908 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.051909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.051920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.051949 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.052820 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.053504 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.054769 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.055073 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.055137 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.056850 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.056851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.056851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.056857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.058584 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.059140 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.059141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.059142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.059212 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.059426 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.059427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.059427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.059585 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.059657 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.059884 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.059953 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:21:31.060079 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.050235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.4.207.62:35991)
0923 22:21:31.050274 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.050276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.050476 (+ 200us) server_negotiation.cc:407] Connection header received
0923 22:21:31.050642 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.050644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.050667 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.050714 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.051720 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.052519 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.053732 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.054036 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.054097 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.056304 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.056305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.056306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.056313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.058368 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.058965 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.058966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.058966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.059034 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.059339 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.059340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.059341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.059503 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.059575 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.059905 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.059994 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 22:21:31.061233 27330 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [term 1 FOLLOWER]: Refusing update from remote peer 3f432b564e0f42a39dac0d8f2b5055ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:31.061573 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.050532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.4.207.61:40465)
0923 22:21:31.050575 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.050577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.050986 (+ 409us) server_negotiation.cc:407] Connection header received
0923 22:21:31.051053 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.051054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.051066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.051135 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.051857 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.052576 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.056370 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.056696 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.056738 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.058589 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.058590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.058591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.058597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.060308 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.060741 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.060742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.060742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.060798 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.061001 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.061002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.061002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.061127 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.061203 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.061403 (+ 200us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.061476 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:31.061547 27388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:31.063256 27362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f432b564e0f42a39dac0d8f2b5055ec. Latest consensus state: current_term: 1 leader_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.063359 27362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:31.072563 27376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.072655 27376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:31.072853 27362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.072932 27362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:31.073173 27400 mvcc.cc:205] Tried to move back new op lower bound from 7332667764984057856 to 7332667764939722752. Current Snapshot: MvccSnapshot[applied={T|T < 7332667764984057856}]
I20260923 22:21:31.080194 27394 catalog_manager.cc:1347] Generated new cluster ID: 76918c0700d34c4c8ec4cb988a31319d
I20260923 22:21:31.080253 27394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:31.084106 27281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } attempt: 3
Sep 23 22:21:31 dist-test-slave-7511 krb5kdc[27125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202090, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:21:31.100116 27360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.084551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.4.207.60:42007)
0923 22:21:31.084880 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.084882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.084889 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:21:31.084909 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.084910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.084919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.085311 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.085341 (+ 30us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.086197 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.087387 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.087726 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.088102 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.090214 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:31.090215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:31.090216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:31.090222 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:31.091792 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.092151 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.092151 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.092152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.092200 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:31.099478 (+ 7278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:31.099480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:31.099480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:31.099629 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:31.099719 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:31.099925 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 22:21:31.100017 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307}
I20260923 22:21:31.100690 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:31.084347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:42007 (local address 127.0.0.1:48208)
0923 22:21:31.084424 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:31.084436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:31.084471 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:31.084997 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:31.084998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:31.085180 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:31.085246 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.085248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.086312 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.086312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.087158 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:31.087160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:31.087823 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:31.087824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:31.087895 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:31.088398 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:31.088403 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:31.090068 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:31.091886 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.091887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.091888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.092079 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.092267 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:31.092268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:31.092269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:31.092342 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:31.099729 (+ 7387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:31.099730 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:31.099787 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 22:21:31.100469 (+ 682us) client_negotiation.cc:240] Negotiation successful
0923 22:21:31.100587 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 22:21:31.102491 27281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976: Bootstrap starting.
I20260923 22:21:31.103219 27281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:31.103551 27281 log.cc:824] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:31.108055 27281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976: No bootstrap required, opened a new log
I20260923 22:21:31.110002 27281 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } }
I20260923 22:21:31.110132 27281 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67c69b979524a6aa0a7a2f214151976, State: Initialized, Role: FOLLOWER
I20260923 22:21:31.110245 27281 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } }
I20260923 22:21:31.110630 27281 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:31.111716 27254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:31.112705 27254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [term 1 FOLLOWER]: Refusing update from remote peer 3f432b564e0f42a39dac0d8f2b5055ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:31.113039 27388 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:31.113369 27255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f67c69b979524a6aa0a7a2f214151976"
I20260923 22:21:31.114209 27255 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f432b564e0f42a39dac0d8f2b5055ec in term 1.
I20260923 22:21:31.115512 27394 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:31.116322 27394 catalog_manager.cc:1504] Loading token signing keys...
W20260923 22:21:31.118445 27421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:31.118552 27421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:31.120769 27413 mvcc.cc:205] Tried to move back new op lower bound from 7332667765193928704 to 7332667764939722752. Current Snapshot: MvccSnapshot[applied={T|T < 7332667764984057856}]
I20260923 22:21:31.121589 27415 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f432b564e0f42a39dac0d8f2b5055ec. Latest consensus state: current_term: 1 leader_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.121745 27415 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:31.121835 27415 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.121937 27415 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:31.127681 27412 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.127799 27412 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:31.128907 27388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f432b564e0f42a39dac0d8f2b5055ec" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 35991 } } peers { permanent_uuid: "f67c69b979524a6aa0a7a2f214151976" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 40465 } } peers { permanent_uuid: "c4fea4e331904944b8de83914964c771" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 42007 } } }
I20260923 22:21:31.129004 27388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:31.130226 27394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f432b564e0f42a39dac0d8f2b5055ec: Generated new TSK 0
I20260923 22:21:31.130388 27394 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 22:21:31.161631 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.160053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.4.207.61:40465)
0923 22:21:31.160095 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.160096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.160894 (+ 798us) server_negotiation.cc:407] Connection header received
0923 22:21:31.161010 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.161012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.161025 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.161057 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.161552 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42262: BlockingRecv error: recv got EOF from 127.0.0.1:42262 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 22:21:31.162424 27360 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.159795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48220 (local address 127.4.207.60:42007)
0923 22:21:31.159843 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.159846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.161884 (+ 2038us) server_negotiation.cc:407] Connection header received
0923 22:21:31.161987 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.161988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.161999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.162030 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.162362 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48220: BlockingRecv error: recv got EOF from 127.0.0.1:48220 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260923 22:21:31.163170 27209 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.160284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.4.207.62:35991)
0923 22:21:31.160323 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.160325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.162669 (+ 2344us) server_negotiation.cc:407] Connection header received
0923 22:21:31.162768 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.162769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.162780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.162809 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.163110 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50592: BlockingRecv error: recv got EOF from 127.0.0.1:50592 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 22:21:31.165647 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.164870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.4.207.61:40465)
0923 22:21:31.164905 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.164906 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.165133 (+ 227us) server_negotiation.cc:407] Connection header received
0923 22:21:31.165254 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.165255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.165273 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.165304 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.165602 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42264: BlockingRecv error: recv got EOF from 127.0.0.1:42264 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 22:21:31.168738 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.167982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42272 (local address 127.4.207.61:40465)
0923 22:21:31.168014 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.168015 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.168280 (+ 265us) server_negotiation.cc:407] Connection header received
0923 22:21:31.168373 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.168374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.168384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.168411 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.168698 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42272: BlockingRecv error: recv got EOF from 127.0.0.1:42272 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 22:21:31.174399 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.173607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.4.207.61:40465)
0923 22:21:31.173642 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.173644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.173893 (+ 249us) server_negotiation.cc:407] Connection header received
0923 22:21:31.173987 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.173988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.173997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.174026 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.174358 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42276: BlockingRecv error: recv got EOF from 127.0.0.1:42276 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 22:21:31.180931 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.180013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.4.207.61:40465)
0923 22:21:31.180051 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.180053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.180347 (+ 294us) server_negotiation.cc:407] Connection header received
0923 22:21:31.180452 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.180453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.180477 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.180506 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.180879 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42286: BlockingRecv error: recv got EOF from 127.0.0.1:42286 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:21:31.191066 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.190162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.4.207.61:40465)
0923 22:21:31.190205 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.190208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.190491 (+ 283us) server_negotiation.cc:407] Connection header received
0923 22:21:31.190604 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.190605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.190617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.190648 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.191010 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42296: BlockingRecv error: recv got EOF from 127.0.0.1:42296 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 22:21:31.209290 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.208415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.4.207.61:40465)
0923 22:21:31.208456 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.208458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.208711 (+ 253us) server_negotiation.cc:407] Connection header received
0923 22:21:31.208811 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.208812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.208823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.208852 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.209230 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42312: BlockingRecv error: recv got EOF from 127.0.0.1:42312 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:21:31.243419 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.242628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.4.207.61:40465)
0923 22:21:31.242679 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.242681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.242841 (+ 160us) server_negotiation.cc:407] Connection header received
0923 22:21:31.242918 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.242919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.242938 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.243003 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.243344 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42314: BlockingRecv error: recv got EOF from 127.0.0.1:42314 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 22:21:31.309561 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.308845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.4.207.61:40465)
0923 22:21:31.308929 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.308930 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.309028 (+ 98us) server_negotiation.cc:407] Connection header received
0923 22:21:31.309099 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.309100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.309110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.309162 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.309479 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59058: BlockingRecv error: recv got EOF from 127.0.0.1:59058 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 22:21:31.439766 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.438882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.4.207.61:40465)
0923 22:21:31.438983 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.438985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.439112 (+ 127us) server_negotiation.cc:407] Connection header received
0923 22:21:31.439238 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.439239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.439249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.439317 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.439710 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59060: BlockingRecv error: recv got EOF from 127.0.0.1:59060 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 22:21:31.697747 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:31.697074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.4.207.61:40465)
0923 22:21:31.697139 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:31.697142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:31.697219 (+ 77us) server_negotiation.cc:407] Connection header received
0923 22:21:31.697265 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:31.697266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:31.697283 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:31.697345 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:31.697704 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59076: BlockingRecv error: recv got EOF from 127.0.0.1:59076 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 22:21:32.040192 27373 catalog_manager.cc:1259] Loaded cluster ID: 76918c0700d34c4c8ec4cb988a31319d
I20260923 22:21:32.040294 27373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771: loading cluster ID for follower catalog manager: success
I20260923 22:21:32.041121 27373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771: acquiring CA information for follower catalog manager: success
I20260923 22:21:32.041494 27373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c4fea4e331904944b8de83914964c771: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:21:32.120656 27421 catalog_manager.cc:1259] Loaded cluster ID: 76918c0700d34c4c8ec4cb988a31319d
I20260923 22:21:32.120790 27421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976: loading cluster ID for follower catalog manager: success
I20260923 22:21:32.121754 27421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976: acquiring CA information for follower catalog manager: success
I20260923 22:21:32.122084 27421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f67c69b979524a6aa0a7a2f214151976: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:21:32.213937 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:32.211083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.4.207.61:40465)
0923 22:21:32.211254 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:32.211256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:32.211294 (+ 38us) server_negotiation.cc:407] Connection header received
0923 22:21:32.211363 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:32.211364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:32.211413 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:32.211514 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:32.211807 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.212350 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.213122 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.213317 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.213375 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:32.213568 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:32.213691 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:32.213751 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:32.213789 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 22:21:32.213808 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":126,"threads_started":1}
I20260923 22:21:32.218078 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:32.215858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52136 (local address 127.4.207.60:42007)
0923 22:21:32.215991 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:32.215993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:32.216001 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:32.216037 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:32.216038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:32.216073 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:32.216140 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:32.216227 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.216682 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.217401 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.217599 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.217649 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:32.217768 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:32.217866 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:32.217919 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:32.218005 (+ 86us) server_negotiation.cc:299] Negotiation successful
0923 22:21:32.218020 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260923 22:21:32.222420 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:32.219757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.4.207.61:40465)
0923 22:21:32.219801 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:32.219803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:32.219915 (+ 112us) server_negotiation.cc:407] Connection header received
0923 22:21:32.219966 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:32.219968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:32.219979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:32.220028 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:32.220168 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.220687 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.221676 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.222010 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.222088 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:32.222187 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:32.222217 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:32.222262 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:32.222328 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 22:21:32.222351 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:21:32.222868 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:32.219716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.4.207.60:42007)
0923 22:21:32.219830 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:32.219832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:32.220007 (+ 175us) server_negotiation.cc:407] Connection header received
0923 22:21:32.220032 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:32.220033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:32.220041 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:32.220092 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:32.220272 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.221008 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.222007 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.222302 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.222359 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:32.222528 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:32.222559 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:32.222602 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:32.222744 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 22:21:32.222764 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260923 22:21:32.223145 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:32.219648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.4.207.62:35991)
0923 22:21:32.219750 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:32.219752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:32.219912 (+ 160us) server_negotiation.cc:407] Connection header received
0923 22:21:32.220053 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:32.220054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:32.220129 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:32.220218 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:32.220319 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.221091 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.221996 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:32.222332 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:32.222461 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:32.222728 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:32.222840 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:32.222882 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:32.223022 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 22:21:32.223062 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":49,"threads_started":1}
I20260923 22:21:32.224947 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 27138
I20260923 22:21:32.230206 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 27213
I20260923 22:21:32.235098 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 27288
2026-09-23T22:21:32Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2800 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 22:21:32 dist-test-slave-7511 krb5kdc[27506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 22:21:32 dist-test-slave-7511 krb5kdc[27506](info): set up 2 sockets
Sep 23 22:21:32 dist-test-slave-7511 krb5kdc[27506](info): commencing operation
krb5kdc: starting...
W20260923 22:21:33.255749 4924 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/test-user.keytab.
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T22:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T22:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.207.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
Entry for principal kudu/127.4.207.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab.
I20260923 22:21:33.369793 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42859
--webserver_interface=127.4.207.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:42859,127.4.207.61:43225,127.4.207.60:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 22:21:33.447430 27519 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:33.447679 27519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:33.447710 27519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:33.447739 27519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:33.448849 27519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:33.448920 27519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:33.448943 27519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:33.448971 27519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:33.449000 27519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:33.450440 27519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:42859,127.4.207.61:43225,127.4.207.60:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.62.keytab
--principal=kudu/127.4.207.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.62:42859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.207.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:33.450718 27519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:33.450940 27519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:33.455875 27527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:33.455869 27525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:33.455879 27524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:33.456176 27519 server_base.cc:1034] running on GCE node
I20260923 22:21:33.456374 27519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:33.456597 27519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:33.457777 27519 hybrid_clock.cc:648] HybridClock initialized: now 1790202093457762 us; error 28 us; skew 500 ppm
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:33.463409 27519 init.cc:377] Logged in from keytab as kudu/127.4.207.62@KRBTEST.COM (short username kudu)
I20260923 22:21:33.463953 27519 webserver.cc:460] Webserver started at http://127.4.207.62:39643/ using document root <none> and password file <none>
I20260923 22:21:33.464236 27519 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:33.464304 27519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:33.464448 27519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:33.465477 27519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/data/instance:
uuid: "bf6fa3c733ae40d19d632e7727868c06"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "b46d00debee2ddde0b228336e95c3947"
server_key_iv: "6d8141c9c73cf3e80917fec6fc333427"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.465829 27519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/wal/instance:
uuid: "bf6fa3c733ae40d19d632e7727868c06"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "b46d00debee2ddde0b228336e95c3947"
server_key_iv: "6d8141c9c73cf3e80917fec6fc333427"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.467115 27519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 22:21:33.467906 27534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:33.468070 27519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:33.468194 27519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/wal
uuid: "bf6fa3c733ae40d19d632e7727868c06"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "b46d00debee2ddde0b228336e95c3947"
server_key_iv: "6d8141c9c73cf3e80917fec6fc333427"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.468266 27519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:33.477154 27519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:33.479357 27519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:33.479534 27519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:33.483742 27519 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.62:42859
I20260923 22:21:33.483846 27586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.62:42859 every 8 connection(s)
I20260923 22:21:33.484169 27519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/data/info.pb
I20260923 22:21:33.485103 27587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:33.486485 27587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:33.494366 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 27519
I20260923 22:21:33.494504 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-0/wal/instance
I20260923 22:21:33.494696 4924 external_mini_cluster.cc:1428] Setting key 9e472af494c8f7f42108a91cc376136d
I20260923 22:21:33.495823 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.487087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42859 (local address 127.0.0.1:47134)
0923 22:21:33.487283 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.487341 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.487852 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.488139 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.488142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.488446 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.488583 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.488587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.489584 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.489585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.490236 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.490238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.490728 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.490730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.490794 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.491251 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.491259 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.492621 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.494243 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.494250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.494252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.494448 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.494762 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.494763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.494763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.494849 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.495198 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.495202 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.495265 (+ 63us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.495378 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.495632 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":99,"thread_start_us":95,"threads_started":1}
I20260923 22:21:33.495829 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.487089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47134 (local address 127.4.207.62:42859)
0923 22:21:33.487292 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.487300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.487463 (+ 163us) server_negotiation.cc:407] Connection header received
0923 22:21:33.488012 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.488019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.488036 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.488135 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.488697 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.489476 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.490319 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.490611 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.490742 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.492716 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.492722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.492723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.492731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.494105 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.494566 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.494588 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.494589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.494685 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.494918 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.494920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.494920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.495118 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.495218 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.495415 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.495568 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260923 22:21:33.496935 27587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:33.497308 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.497166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51666)
0923 22:21:33.497220 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.497272 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 22:21:33.497467 27535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
I20260923 22:21:33.497936 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.497812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51676)
0923 22:21:33.497868 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.497904 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 22:21:33.498098 27587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:43225: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.207.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab.
Entry for principal kudu/127.4.207.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab.
I20260923 22:21:33.518676 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.61:43225
--webserver_interface=127.4.207.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab
--principal=kudu/127.4.207.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:42859,127.4.207.61:43225,127.4.207.60:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:21:33.547369 27587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } attempt: 1
I20260923 22:21:33.548010 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.547844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51686)
0923 22:21:33.547921 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.547971 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 22:21:33.548436 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.548340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51688)
0923 22:21:33.548387 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.548416 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260923 22:21:33.548581 27587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:43225: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
W20260923 22:21:33.599848 27594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:33.600051 27594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:33.600069 27594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:33.600090 27594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:33.601248 27594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:33.601289 27594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:33.601305 27594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:33.601316 27594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:33.601366 27594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:33.602869 27594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:42859,127.4.207.61:43225,127.4.207.60:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.61.keytab
--principal=kudu/127.4.207.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.61:43225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.207.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:33.603116 27594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:33.603493 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:33.609124 27600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:33.609061 27599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:33.609233 27594 server_base.cc:1034] running on GCE node
W20260923 22:21:33.609061 27602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:33.609731 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:33.609973 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:33.611127 27594 hybrid_clock.cc:648] HybridClock initialized: now 1790202093611110 us; error 32 us; skew 500 ppm
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:33.613663 27594 init.cc:377] Logged in from keytab as kudu/127.4.207.61@KRBTEST.COM (short username kudu)
I20260923 22:21:33.614079 27594 webserver.cc:460] Webserver started at http://127.4.207.61:40041/ using document root <none> and password file <none>
I20260923 22:21:33.614311 27594 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:33.614355 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:33.614506 27594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:33.615566 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/data/instance:
uuid: "400ba0fd9f6c419dabf98a34140fb142"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "60be74ba7313f86f7e30764f84bd8aff"
server_key_iv: "096da4111e490d5ce307e27e92f2d9b6"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.615919 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/wal/instance:
uuid: "400ba0fd9f6c419dabf98a34140fb142"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "60be74ba7313f86f7e30764f84bd8aff"
server_key_iv: "096da4111e490d5ce307e27e92f2d9b6"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.616767 27587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } attempt: 2
I20260923 22:21:33.617172 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.617017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51692)
0923 22:21:33.617086 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.617130 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 22:21:33.617352 27594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 22:21:33.617583 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.617498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51706)
0923 22:21:33.617531 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.617565 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260923 22:21:33.617671 27587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.61:43225: Network error: Client connection negotiation failed: client connection to 127.4.207.61:43225: connect: Connection refused (error 111)
I20260923 22:21:33.618198 27609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:33.618345 27594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 22:21:33.618422 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/wal
uuid: "400ba0fd9f6c419dabf98a34140fb142"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "60be74ba7313f86f7e30764f84bd8aff"
server_key_iv: "096da4111e490d5ce307e27e92f2d9b6"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.618477 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:33.649495 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:33.651306 27594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:33.651415 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:33.655376 27594 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.61:43225
I20260923 22:21:33.655498 27661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.61:43225 every 8 connection(s)
I20260923 22:21:33.655810 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/data/info.pb
I20260923 22:21:33.656785 27662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:33.658310 27662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:33.663553 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 27594
I20260923 22:21:33.663673 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-1/wal/instance
I20260923 22:21:33.663846 4924 external_mini_cluster.cc:1428] Setting key 4a945e905939d245541a5c65ae97a0d5
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:33.667757 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.659003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.4.207.62:42859)
0923 22:21:33.659046 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.659048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.659278 (+ 230us) server_negotiation.cc:407] Connection header received
0923 22:21:33.659832 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.659834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.659849 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.659904 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.660511 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.661016 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.662013 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.662227 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.662324 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.664603 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.664604 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.664605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.664613 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.666126 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.666656 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.666657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.666657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.666718 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.667000 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.667001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.667001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.667094 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.667141 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.667536 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.667634 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 22:21:33.668044 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.658973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42859 (local address 127.0.0.1:47150)
0923 22:21:33.659154 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.659187 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.659722 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.659987 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.659996 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.660232 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.660414 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.660418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.661114 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.661116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.661907 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.661909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.662344 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.662345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.662460 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.662904 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.662918 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.664514 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.666213 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.666221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.666223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.666573 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.666783 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.666784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.666785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.666934 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.667167 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.667173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.667349 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.667526 (+ 177us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.667750 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260923 22:21:33.668638 27662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.207.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.207.60@KRBTEST.COM" created.
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:21:33.679313 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.668928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51708)
0923 22:21:33.669179 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.669192 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.669253 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.669408 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.669409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.669578 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.669644 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.669646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.670606 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.670608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.671593 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.671596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.672173 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.672174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.672254 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.672596 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.672602 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.676189 (+ 3587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.678037 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.678038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.678038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.678298 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.678504 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.678505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.678505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.678584 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.678916 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.678917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.678970 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.679059 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.679182 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260923 22:21:33.679332 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.669007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.4.207.61:43225)
0923 22:21:33.669152 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.669156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.669245 (+ 89us) server_negotiation.cc:407] Connection header received
0923 22:21:33.669330 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.669331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.669341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.669400 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.669719 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.670493 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.671704 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.672078 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.672133 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.676282 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.676291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.676291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.676299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.677946 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.678386 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.678388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.678389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.678436 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.678656 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.678656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.678657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.678839 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.678904 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.679090 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.679252 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":124,"thread_start_us":103,"threads_started":1}
I20260923 22:21:33.680068 27662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:33.680456 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.680311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36706)
0923 22:21:33.680366 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.680422 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 22:21:33.680614 27610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
I20260923 22:21:33.681070 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.680957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36712)
0923 22:21:33.681009 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.681043 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 22:21:33.681237 27662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:45583: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.207.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab.
Entry for principal kudu/127.4.207.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab.
I20260923 22:21:33.690280 4924 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMonjHj/build/release/bin/kudu
/tmp/dist-test-taskMonjHj/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.60:45583
--webserver_interface=127.4.207.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab
--principal=kudu/127.4.207.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.207.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.207.62:42859,127.4.207.61:43225,127.4.207.60:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 22:21:33.708825 27587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } attempt: 3
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:21:33.715977 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.709190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51716)
0923 22:21:33.709252 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.709262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.709315 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.709467 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.709468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.709621 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.709682 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.709684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.710375 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.710376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.710892 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.710894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.711311 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.711312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.711393 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.711944 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.711949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.713212 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.714829 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.714830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.714831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.715002 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.715221 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.715222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.715223 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.715290 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.715629 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.715630 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.715679 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.715774 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.715879 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 22:21:33.715977 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.709247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.4.207.61:43225)
0923 22:21:33.709307 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.709309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.709318 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:21:33.709382 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.709382 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.709400 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.709462 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.709795 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.710293 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.711009 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.711238 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.711314 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.713321 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.713322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.713323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.713332 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.714722 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.715088 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.715089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.715089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.715152 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.715364 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.715365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.715365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.715558 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.715651 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.715800 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.715881 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 22:21:33.716318 27587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:33.716693 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.716573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36714)
0923 22:21:33.716624 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.716661 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:21:33.717083 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.716975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36722)
0923 22:21:33.717019 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.717045 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:21:33.717202 27587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:45583: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
I20260923 22:21:33.730472 27662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } attempt: 1
I20260923 22:21:33.730949 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.730798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36738)
0923 22:21:33.730879 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.730919 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 22:21:33.731300 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.731207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36744)
0923 22:21:33.731252 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.731278 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260923 22:21:33.731428 27662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:45583: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
I20260923 22:21:33.748358 27587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } attempt: 1
I20260923 22:21:33.748790 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.748652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36758)
0923 22:21:33.748723 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.748761 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 22:21:33.749132 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.749041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36766)
0923 22:21:33.749084 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.749108 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 22:21:33.749243 27587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:45583: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
W20260923 22:21:33.770968 27669 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 22:21:33.771202 27669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 22:21:33.771242 27669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 22:21:33.771284 27669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 22:21:33.772460 27669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 22:21:33.772524 27669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 22:21:33.772550 27669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 22:21:33.772581 27669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 22:21:33.772610 27669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 22:21:33.774005 27669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.207.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.207.62:42859,127.4.207.61:43225,127.4.207.60:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/krb5kdc/kudu_127.4.207.60.keytab
--principal=kudu/127.4.207.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.207.60:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.207.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 21:02:32 UTC on ccycloud.quasar-lecndr.root.comops.site
build id 4668598
I20260923 22:21:33.774271 27669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 22:21:33.774483 27669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 22:21:33.779150 27674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 22:21:33.779181 27675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:33.779407 27669 server_base.cc:1034] running on GCE node
W20260923 22:21:33.779520 27677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 22:21:33.779840 27669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 22:21:33.780139 27669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 22:21:33.781332 27669 hybrid_clock.cc:648] HybridClock initialized: now 1790202093781324 us; error 41 us; skew 500 ppm
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 22:21:33.783834 27669 init.cc:377] Logged in from keytab as kudu/127.4.207.60@KRBTEST.COM (short username kudu)
I20260923 22:21:33.784192 27669 webserver.cc:460] Webserver started at http://127.4.207.60:44163/ using document root <none> and password file <none>
I20260923 22:21:33.784457 27669 fs_manager.cc:359] Metadata directory not provided
I20260923 22:21:33.784513 27669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 22:21:33.784597 27669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 22:21:33.785573 27669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/data/instance:
uuid: "94e4e4651fc14481ab602e50d1d01be7"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "12eb1b9692788081e244535d2eeb94c3"
server_key_iv: "47e3903d410637c435edba719690155e"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.785884 27669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/wal/instance:
uuid: "94e4e4651fc14481ab602e50d1d01be7"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "12eb1b9692788081e244535d2eeb94c3"
server_key_iv: "47e3903d410637c435edba719690155e"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.787104 27669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 22:21:33.787865 27684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:33.788045 27669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 22:21:33.788144 27669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/data,/tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/wal
uuid: "94e4e4651fc14481ab602e50d1d01be7"
format_stamp: "Formatted at 2026-09-23 22:21:33 on dist-test-slave-7511"
server_key: "12eb1b9692788081e244535d2eeb94c3"
server_key_iv: "47e3903d410637c435edba719690155e"
server_key_version: "encryptionkey@0"
I20260923 22:21:33.788223 27669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 22:21:33.796763 27669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 22:21:33.798403 27669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 22:21:33.798561 27669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 22:21:33.799579 27662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } attempt: 2
I20260923 22:21:33.799971 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.799819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36778)
0923 22:21:33.799886 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.799929 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 22:21:33.800383 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.800256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36780)
0923 22:21:33.800320 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.800351 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260923 22:21:33.800486 27662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.207.60:45583: Network error: Client connection negotiation failed: client connection to 127.4.207.60:45583: connect: Connection refused (error 111)
I20260923 22:21:33.802577 27669 rpc_server.cc:307] RPC server started. Bound to: 127.4.207.60:45583
I20260923 22:21:33.802620 27736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.207.60:45583 every 8 connection(s)
I20260923 22:21:33.802946 27669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/data/info.pb
I20260923 22:21:33.803957 27737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 22:21:33.804756 4924 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMonjHj/build/release/bin/kudu as pid 27669
I20260923 22:21:33.804893 4924 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMonjHj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790201827590652-4924-0/minicluster-data/master-2/wal/instance
I20260923 22:21:33.805089 4924 external_mini_cluster.cc:1428] Setting key 38c131bcb852aaabc86e797704c1bee9
I20260923 22:21:33.805742 27737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:21:33.815342 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.806723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.4.207.62:42859)
0923 22:21:33.806772 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.806774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.806882 (+ 108us) server_negotiation.cc:407] Connection header received
0923 22:21:33.806953 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.806954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.806963 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.807014 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.807602 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.808349 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.809569 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.809864 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.809937 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.812414 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.812415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.812415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.812422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.813926 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.814398 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.814399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.814400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.814446 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.814724 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.814725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.814725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.814818 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.814894 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.815172 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.815234 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 22:21:33.815743 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.806599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.62:42859 (local address 127.0.0.1:47156)
0923 22:21:33.806766 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.806808 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.806881 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.807044 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.807064 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.807365 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.807519 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.807524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.808425 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.808426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.809482 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.809484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.809965 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.809966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.810049 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.810535 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.810543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.812244 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.814004 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.814011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.814013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.814322 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.814514 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.814515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.814515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.814654 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.814886 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.814891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.815022 (+ 131us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.815135 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.815294 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":112,"threads_started":1}
I20260923 22:21:33.816335 27737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:33.816407 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.805560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.4.207.60:45583)
0923 22:21:33.805742 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.805751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.805781 (+ 30us) server_negotiation.cc:407] Connection header received
0923 22:21:33.806224 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.806231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.806275 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.806422 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.807202 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.808534 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.809568 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.810049 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.810109 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.812808 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.812816 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.812816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.812824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.814817 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.815156 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.815159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.815160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.815214 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.815412 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.815413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.815413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.815743 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.815889 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.816183 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.816277 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.62@KRBTEST.COM
I20260923 22:21:33.823168 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.816694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.61:43225 (local address 127.0.0.1:51728)
0923 22:21:33.816745 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.816757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.816796 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.816943 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.816944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.817070 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.817128 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.817130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.817800 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.817801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.818261 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.818262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.818625 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.818626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.818684 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.819083 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.819086 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.820231 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.822126 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.822127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.822128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.822366 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.822577 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.822578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.822578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.822657 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.822955 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.822955 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.822989 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.823047 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.823112 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 22:21:33.823238 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.816784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.4.207.61:43225)
0923 22:21:33.816823 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.816824 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.816834 (+ 10us) server_negotiation.cc:407] Connection header received
0923 22:21:33.816863 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.816864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.816871 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.816938 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.817199 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.817713 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.818353 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.818548 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.818624 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.820359 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.820360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.820361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.820368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.822055 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.822449 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.822450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.822450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.822507 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.822738 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.822739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.822739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.822884 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.822957 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.823092 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.823159 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 22:21:33.823757 27737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } has no permanent_uuid. Determining permanent_uuid...
I20260923 22:21:33.824376 27587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } attempt: 2
I20260923 22:21:33.825111 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.817414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.4.207.62:42859)
0923 22:21:33.817449 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.817451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.817469 (+ 18us) server_negotiation.cc:407] Connection header received
0923 22:21:33.817489 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.817490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.817499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.817556 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.818016 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.818475 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.819325 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.819582 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.819656 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.821853 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.821855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.821855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.821862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.823649 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.824039 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.824040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.824041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.824096 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.824309 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.824309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.824309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.824441 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.824914 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.824953 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.825019 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.60@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.62@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.207.61@KRBTEST.COM
I20260923 22:21:33.831135 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.824049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.4.207.60:45583)
0923 22:21:33.824091 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.824093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.824101 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:33.824144 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.824145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.824156 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.824202 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.824469 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.825163 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.826176 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.826435 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.826507 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.828349 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.828350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.828351 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.828356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.829998 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.830353 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.830354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.830354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.830409 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.830582 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.830583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.830583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.830700 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.830773 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.830961 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.831028 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 22:21:33.831034 27741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.824019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36802)
0923 22:21:33.824048 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.824056 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.824092 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.824207 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.824208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.824357 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.824389 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.824391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.825241 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.825241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.826095 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.826097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.826516 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.826516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.826572 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.826963 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.826969 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.828277 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.830090 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.830092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.830092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.830283 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.830480 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.830481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.830481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.830528 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.830782 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.830783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.830829 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.830891 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.830980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260923 22:21:33.832719 27737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7: Bootstrap starting.
I20260923 22:21:33.833036 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.824582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36806)
0923 22:21:33.824632 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.824644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.824690 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.824936 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.824937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.825085 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.825122 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.825123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.826098 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.826099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.826962 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.826965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.827427 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.827428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.827507 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.827889 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.827893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.829289 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.831898 (+ 2609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.831899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.831900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.832102 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.832382 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.832384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.832384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.832442 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.832748 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.832750 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.832795 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.832845 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.832938 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 22:21:33.833372 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.824657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.4.207.60:45583)
0923 22:21:33.824802 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.824804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.824811 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:21:33.824828 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.824829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.824837 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.825280 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.825307 (+ 27us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.826019 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.827053 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.827354 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.827427 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.829385 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.829386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.829386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.829392 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.831800 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.832199 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.832200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.832200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.832282 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.832515 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.832515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.832515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.832663 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.833177 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.833216 (+ 39us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.833287 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260923 22:21:33.833534 27737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:33.833868 27737 log.cc:824] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:33.834659 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.825660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.4.207.61:43225)
0923 22:21:33.825702 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.825704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.825712 (+ 8us) server_negotiation.cc:407] Connection header received
0923 22:21:33.825767 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.825767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.825776 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.825830 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.826414 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.827121 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.828125 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.828435 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.828504 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.831111 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.831112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.831113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.831120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.832890 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.833710 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.833711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.833711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.833771 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.834005 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.834006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.834006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.834137 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.834251 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.834452 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.834542 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:21:33.834695 27737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7: No bootstrap required, opened a new log
I20260923 22:21:33.835131 27587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06: Bootstrap starting.
I20260923 22:21:33.835770 27587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:33.836058 27587 log.cc:824] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:33.836237 27737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } }
I20260923 22:21:33.836350 27737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94e4e4651fc14481ab602e50d1d01be7, State: Initialized, Role: FOLLOWER
I20260923 22:21:33.836452 27737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } }
I20260923 22:21:33.836736 27745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.836809 27737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:33.836802 27587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06: No bootstrap required, opened a new log
I20260923 22:21:33.836836 27745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 22:21:33.838717 27756 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:33.838776 27756 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:33.839318 4924 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 22:21:33.839277 27587 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } }
I20260923 22:21:33.839363 27587 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6fa3c733ae40d19d632e7727868c06, State: Initialized, Role: FOLLOWER
I20260923 22:21:33.839409 27587 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } }
I20260923 22:21:33.839674 27758 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.839749 27587 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:33.839751 27758 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:33.840370 27561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bf6fa3c733ae40d19d632e7727868c06"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47158
I20260923 22:21:33.840480 27561 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 22:21:33.840525 27561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:33.838877 27669 master_runner.cc:186] Error getting master registration for 127.4.207.62:42859: OK, instance_id { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" instance_seqno: 1790202093482485 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:21:33.841302 27669 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:21:33.841307 27561 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } }
I20260923 22:21:33.841538 27561 leader_election.cc:288] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [CANDIDATE]: Term 1 election: Requested vote from peers 400ba0fd9f6c419dabf98a34140fb142 (127.4.207.61:43225), 94e4e4651fc14481ab602e50d1d01be7 (127.4.207.60:45583)
I20260923 22:21:33.841871 27711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf6fa3c733ae40d19d632e7727868c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94e4e4651fc14481ab602e50d1d01be7"
I20260923 22:21:33.841979 27711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:33.842808 27711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf6fa3c733ae40d19d632e7727868c06 in term 1.
I20260923 22:21:33.843149 27535 leader_election.cc:302] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94e4e4651fc14481ab602e50d1d01be7, bf6fa3c733ae40d19d632e7727868c06; no voters:
I20260923 22:21:33.843274 27758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 22:21:33.843405 27758 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [term 1 LEADER]: Becoming Leader. State: Replica: bf6fa3c733ae40d19d632e7727868c06, State: Running, Role: LEADER
I20260923 22:21:33.843540 27758 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } }
I20260923 22:21:33.843927 27758 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf6fa3c733ae40d19d632e7727868c06. Latest consensus state: current_term: 1 leader_uuid: "bf6fa3c733ae40d19d632e7727868c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.844004 27758 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:33.844317 27519 master_runner.cc:186] Error getting master registration for 127.4.207.61:43225: OK, instance_id { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" instance_seqno: 1790202093654318 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 22:21:33.846753 27519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 22:21:33.852022 27779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 22:21:33.852170 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.843044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.4.207.62:42859)
0923 22:21:33.843501 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.843503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.843512 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:21:33.843619 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.843620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.843629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.843659 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.844591 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.845337 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.846324 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.846627 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.846698 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.848336 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.848338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.848338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.848345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.850162 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.850777 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.850779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.850779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.850854 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.851580 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.851581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.851581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.851714 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.851802 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.852024 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.852098 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260923 22:21:33.852546 27779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 22:21:33.855391 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.843825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.4.207.60:45583)
0923 22:21:33.843869 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.843871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.847107 (+ 3236us) server_negotiation.cc:407] Connection header received
0923 22:21:33.847213 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.847214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.847236 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.847266 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.848374 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.849340 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.850249 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.850610 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.850657 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.852205 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.852207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.852207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.852215 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.854059 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.854538 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.854539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.854539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.854591 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.854784 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.854785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.854785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.854905 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.855008 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.855201 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.855283 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:33.855710 27711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [term 1 FOLLOWER]: Refusing update from remote peer bf6fa3c733ae40d19d632e7727868c06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:33.856146 27758 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 22:21:33.856293 27666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.843736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.4.207.61:43225)
0923 22:21:33.843783 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.843786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.846916 (+ 3130us) server_negotiation.cc:407] Connection header received
0923 22:21:33.847024 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.847026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.847038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.847091 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.848341 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.849109 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.850091 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.850808 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.850856 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.852766 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.852768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.852768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.852776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.854501 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.855059 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.855060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.855061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.855116 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.855388 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.855390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.855390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.855544 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.855618 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.856117 (+ 499us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.856197 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 22:21:33.857491 27745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf6fa3c733ae40d19d632e7727868c06. Latest consensus state: current_term: 1 leader_uuid: "bf6fa3c733ae40d19d632e7727868c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.857589 27745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:33.858294 27758 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf6fa3c733ae40d19d632e7727868c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.858390 27758 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:33.858623 27779 catalog_manager.cc:1347] Generated new cluster ID: 412f4a0d91614ca1ba1c87897b8d21b7
I20260923 22:21:33.858672 27779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 22:21:33.858685 27745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf6fa3c733ae40d19d632e7727868c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.858767 27745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:33.858946 27781 mvcc.cc:205] Tried to move back new op lower bound from 7332667776429953024 to 7332667776383844352. Current Snapshot: MvccSnapshot[applied={T|T < 7332667776429953024 or (T in {7332667776429953024})}]
I20260923 22:21:33.859010 27782 mvcc.cc:205] Tried to move back new op lower bound from 7332667776429953024 to 7332667776383844352. Current Snapshot: MvccSnapshot[applied={T|T < 7332667776429953024}]
I20260923 22:21:33.881237 27779 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 22:21:33.881973 27779 catalog_manager.cc:1504] Loading token signing keys...
I20260923 22:21:33.886811 27779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06: Generated new TSK 0
I20260923 22:21:33.887001 27779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 22:21:33.891603 27662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } attempt: 3
Sep 23 22:21:33 dist-test-slave-7511 krb5kdc[27506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790202093, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.207.61@KRBTEST.COM for kudu/127.4.207.60@KRBTEST.COM
I20260923 22:21:33.899581 27664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.891880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.207.60:45583 (local address 127.0.0.1:36836)
0923 22:21:33.891929 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 22:21:33.891940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 22:21:33.891979 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 22:21:33.892124 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 22:21:33.892125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 22:21:33.892323 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0923 22:21:33.892377 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.892379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.893289 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.893290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.894106 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 22:21:33.894108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.894653 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.894654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 22:21:33.894727 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.895215 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 22:21:33.895220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 22:21:33.896824 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 22:21:33.898534 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.898535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.898536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.898753 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.898944 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 22:21:33.898944 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 22:21:33.898945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 22:21:33.898998 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 22:21:33.899260 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 22:21:33.899261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 22:21:33.899304 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 22:21:33.899365 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 22:21:33.899482 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 22:21:33.900103 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:33.891939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.4.207.60:45583)
0923 22:21:33.891975 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.891977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.891986 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:21:33.892034 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.892035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.892046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.892102 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.892456 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.893223 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.894197 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:33.894504 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:33.894594 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:33.896919 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 22:21:33.896920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 22:21:33.896921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 22:21:33.896929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 22:21:33.898445 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.898829 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.898830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.898830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.898881 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 22:21:33.899059 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 22:21:33.899060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 22:21:33.899060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 22:21:33.899177 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 22:21:33.899236 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:33.899939 (+ 703us) server_negotiation.cc:299] Negotiation successful
0923 22:21:33.900015 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 22:21:33.901947 27662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142: Bootstrap starting.
I20260923 22:21:33.902689 27662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142: Neither blocks nor log segments found. Creating new log.
I20260923 22:21:33.902993 27662 log.cc:824] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142: Log is configured to *not* fsync() on all Append() calls
I20260923 22:21:33.903822 27662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142: No bootstrap required, opened a new log
I20260923 22:21:33.905838 27662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } }
I20260923 22:21:33.905985 27662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400ba0fd9f6c419dabf98a34140fb142, State: Initialized, Role: FOLLOWER
I20260923 22:21:33.906064 27662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } }
I20260923 22:21:33.906374 27662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 22:21:33.906692 27636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf6fa3c733ae40d19d632e7727868c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "400ba0fd9f6c419dabf98a34140fb142"
I20260923 22:21:33.906910 27635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [term 0 FOLLOWER]: Advancing to term 1
I20260923 22:21:33.908057 27635 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [term 1 FOLLOWER]: Refusing update from remote peer bf6fa3c733ae40d19d632e7727868c06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 22:21:33.908524 27776 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 22:21:33.908986 27799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 22:21:33.909039 27799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 22:21:33.910555 27801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf6fa3c733ae40d19d632e7727868c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.911541 27801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:33.911645 27801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf6fa3c733ae40d19d632e7727868c06. Latest consensus state: current_term: 1 leader_uuid: "bf6fa3c733ae40d19d632e7727868c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.911751 27801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:33.911821 27801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf6fa3c733ae40d19d632e7727868c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:33.911911 27801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 22:21:33.915998 27802 mvcc.cc:205] Tried to move back new op lower bound from 7332667776556818432 to 7332667776383844352. Current Snapshot: MvccSnapshot[applied={T|T < 7332667776429953024}]
I20260923 22:21:33.916617 27636 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bf6fa3c733ae40d19d632e7727868c06 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260923 22:21:33.932144 27589 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.930469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.4.207.62:42859)
0923 22:21:33.930514 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.930516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.931474 (+ 958us) server_negotiation.cc:407] Connection header received
0923 22:21:33.931584 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.931586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.931598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.931629 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.932069 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47178: BlockingRecv error: recv got EOF from 127.0.0.1:47178 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 22:21:33.933080 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.930867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51772 (local address 127.4.207.61:43225)
0923 22:21:33.930903 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.930905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.932600 (+ 1695us) server_negotiation.cc:407] Connection header received
0923 22:21:33.932692 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.932693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.932704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.932732 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.933024 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51772: BlockingRecv error: recv got EOF from 127.0.0.1:51772 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:21:33.933912 27742 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.931166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.4.207.60:45583)
0923 22:21:33.931206 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.931208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.933408 (+ 2200us) server_negotiation.cc:407] Connection header received
0923 22:21:33.933532 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.933533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.933544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.933572 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.933855 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36852: BlockingRecv error: recv got EOF from 127.0.0.1:36852 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 22:21:33.936138 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.935515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51774 (local address 127.4.207.61:43225)
0923 22:21:33.935544 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.935545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.935702 (+ 157us) server_negotiation.cc:407] Connection header received
0923 22:21:33.935792 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.935793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.935809 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.935835 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.936099 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51774: BlockingRecv error: recv got EOF from 127.0.0.1:51774 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260923 22:21:33.939128 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.938495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.4.207.61:43225)
0923 22:21:33.938524 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.938526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.938717 (+ 191us) server_negotiation.cc:407] Connection header received
0923 22:21:33.938800 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.938801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.938809 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.938834 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.939091 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51776: BlockingRecv error: recv got EOF from 127.0.0.1:51776 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260923 22:21:33.942994 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.942356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.4.207.61:43225)
0923 22:21:33.942386 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.942388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.942568 (+ 180us) server_negotiation.cc:407] Connection header received
0923 22:21:33.942653 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.942654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.942663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.942689 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.942956 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51786: BlockingRecv error: recv got EOF from 127.0.0.1:51786 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260923 22:21:33.948868 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.948179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.4.207.61:43225)
0923 22:21:33.948211 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.948213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.948404 (+ 191us) server_negotiation.cc:407] Connection header received
0923 22:21:33.948490 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.948491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.948500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.948526 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.948829 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51798: BlockingRecv error: recv got EOF from 127.0.0.1:51798 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 22:21:33.959048 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.958152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.4.207.61:43225)
0923 22:21:33.958191 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.958192 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.958476 (+ 284us) server_negotiation.cc:407] Connection header received
0923 22:21:33.958580 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.958581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.958592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.958622 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.958996 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51800: BlockingRecv error: recv got EOF from 127.0.0.1:51800 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 22:21:33.977883 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:33.976943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51812 (local address 127.4.207.61:43225)
0923 22:21:33.976988 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:33.976990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:33.977261 (+ 271us) server_negotiation.cc:407] Connection header received
0923 22:21:33.977374 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:33.977375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:33.977388 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:33.977419 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:33.977820 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51812: BlockingRecv error: recv got EOF from 127.0.0.1:51812 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 22:21:34.012395 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:34.011440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.4.207.61:43225)
0923 22:21:34.011525 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.011527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.011801 (+ 274us) server_negotiation.cc:407] Connection header received
0923 22:21:34.011902 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.011903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.011914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.011944 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:34.012334 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51826: BlockingRecv error: recv got EOF from 127.0.0.1:51826 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 22:21:34.078787 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:34.077930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.4.207.61:43225)
0923 22:21:34.078003 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.078005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.078125 (+ 120us) server_negotiation.cc:407] Connection header received
0923 22:21:34.078251 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.078252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.078262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.078304 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:34.078721 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51828: BlockingRecv error: recv got EOF from 127.0.0.1:51828 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260923 22:21:34.208829 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:34.208078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51842 (local address 127.4.207.61:43225)
0923 22:21:34.208162 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.208164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.208284 (+ 120us) server_negotiation.cc:407] Connection header received
0923 22:21:34.208362 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.208363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.208375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.208431 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:34.208768 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51842: BlockingRecv error: recv got EOF from 127.0.0.1:51842 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260923 22:21:34.466667 27666 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 22:21:34.465967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.4.207.61:43225)
0923 22:21:34.466066 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.466068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.466117 (+ 49us) server_negotiation.cc:407] Connection header received
0923 22:21:34.466164 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.466164 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.466174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.466240 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 22:21:34.466584 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51856: BlockingRecv error: recv got EOF from 127.0.0.1:51856 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 22:21:34.531416 27861 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf6fa3c733ae40d19d632e7727868c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf6fa3c733ae40d19d632e7727868c06" member_type: VOTER last_known_addr { host: "127.4.207.62" port: 42859 } } peers { permanent_uuid: "400ba0fd9f6c419dabf98a34140fb142" member_type: VOTER last_known_addr { host: "127.4.207.61" port: 43225 } } peers { permanent_uuid: "94e4e4651fc14481ab602e50d1d01be7" member_type: VOTER last_known_addr { host: "127.4.207.60" port: 45583 } } }
I20260923 22:21:34.531613 27861 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf6fa3c733ae40d19d632e7727868c06 [sys.catalog]: This master's current role is: LEADER
I20260923 22:21:34.839246 27756 catalog_manager.cc:1259] Loaded cluster ID: 412f4a0d91614ca1ba1c87897b8d21b7
I20260923 22:21:34.839355 27756 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7: loading cluster ID for follower catalog manager: success
I20260923 22:21:34.840386 27756 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7: acquiring CA information for follower catalog manager: success
I20260923 22:21:34.840736 27756 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94e4e4651fc14481ab602e50d1d01be7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:21:34.909464 27799 catalog_manager.cc:1259] Loaded cluster ID: 412f4a0d91614ca1ba1c87897b8d21b7
I20260923 22:21:34.909540 27799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142: loading cluster ID for follower catalog manager: success
I20260923 22:21:34.910370 27799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142: acquiring CA information for follower catalog manager: success
I20260923 22:21:34.910710 27799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 400ba0fd9f6c419dabf98a34140fb142: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 22:21:34.982599 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:34.980096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.4.207.61:43225)
0923 22:21:34.980224 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.980245 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.980254 (+ 9us) server_negotiation.cc:407] Connection header received
0923 22:21:34.980282 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.980282 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.980331 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.980407 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:34.980551 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.981048 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.981829 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.982034 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.982090 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:34.982263 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:34.982367 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:34.982425 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:34.982463 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 22:21:34.982481 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260923 22:21:34.986603 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:34.984463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.4.207.60:45583)
0923 22:21:34.984568 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.984570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.984577 (+ 7us) server_negotiation.cc:407] Connection header received
0923 22:21:34.984613 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.984613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.984647 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.984719 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:34.984794 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.985252 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.985924 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.986108 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.986158 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:34.986296 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:34.986389 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:34.986443 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:34.986475 (+ 32us) server_negotiation.cc:299] Negotiation successful
0923 22:21:34.986491 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260923 22:21:34.991127 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:34.988294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.4.207.61:43225)
0923 22:21:34.988366 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.988368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.988544 (+ 176us) server_negotiation.cc:407] Connection header received
0923 22:21:34.988673 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.988675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.988684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.988736 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:34.988910 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.989459 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.990306 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.990592 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.990657 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:34.990817 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:34.990846 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:34.990888 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:34.991033 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 22:21:34.991054 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 22:21:34.991705 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:34.988525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.4.207.60:45583)
0923 22:21:34.988569 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.988571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.988660 (+ 89us) server_negotiation.cc:407] Connection header received
0923 22:21:34.988791 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.988792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.988804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.988860 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:34.989023 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.989721 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.990699 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.991006 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.991042 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:34.991398 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:34.991445 (+ 47us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:34.991500 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:34.991610 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 22:21:34.991637 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 22:21:34.991953 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 22:21:34.988352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.4.207.62:42859)
0923 22:21:34.988465 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0923 22:21:34.988467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 22:21:34.988771 (+ 304us) server_negotiation.cc:407] Connection header received
0923 22:21:34.988823 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 22:21:34.988824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 22:21:34.988890 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 22:21:34.988989 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 22:21:34.989099 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.989840 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.990836 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 22:21:34.991057 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 22:21:34.991133 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 22:21:34.991510 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 22:21:34.991612 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 22:21:34.991668 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 22:21:34.991811 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 22:21:34.991836 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260923 22:21:34.993345 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 27519
I20260923 22:21:34.998631 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 27594
I20260923 22:21:35.003407 4924 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMonjHj/build/release/bin/kudu with pid 27669
2026-09-23T22:21:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2767 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5568 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267418 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST